2023-11-25T04:43:33.934 INFO:root:teuthology version: 0.0.1.dev205+g2442239 2023-11-25T04:43:33.935 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} --verbose --archive /home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196 --name yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196/orig.config.yaml 2023-11-25T04:43:33.984 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-25T04:43:34.067 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196 branch: main description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false job_id: '7466196' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: ec928368edcc5fc71884a56fd9fd821b148f4e30 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: ec928368edcc5fc71884a56fd9fd821b148f4e30 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: ec928368edcc5fc71884a56fd9fd821b148f4e30 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 77 sha1: ec928368edcc5fc71884a56fd9fd821b148f4e30 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ec928368edcc5fc71884a56fd9fd821b148f4e30/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ec928368edcc5fc71884a56fd9fd821b148f4e30 targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIQLzUj5DoDXok6ckR54zrIrE0YNr5/E8OHdWoHANIrAr9UBNH2fLFLMlj9n4BT4u5otCq4Gq1yMp/4oCqYCggY= smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFtIGrbVuXwbnaMq4SVU04AhZvexOMXUoe+9Sjmpom43axHDTqNK2H6t0ELoI7LvcVNKRAUXDUS45FAfH/4JmEE= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 2442239d2653456406c25ae0c71b689c8f2657b6 timestamp: 2023-11-24_21:18:30 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-11-25T04:43:34.068 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ec928368edcc5fc71884a56fd9fd821b148f4e30/qa; will attempt to use it 2023-11-25T04:43:34.068 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ec928368edcc5fc71884a56fd9fd821b148f4e30/qa/tasks 2023-11-25T04:43:34.069 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-25T04:43:34.070 INFO:teuthology.task.internal:Checking packages... 2023-11-25T04:43:34.097 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ec928368edcc5fc71884a56fd9fd821b148f4e30' 2023-11-25T04:43:34.097 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-25T04:43:34.097 INFO:teuthology.packaging:ref: None 2023-11-25T04:43:34.097 INFO:teuthology.packaging:tag: None 2023-11-25T04:43:34.098 INFO:teuthology.packaging:branch: main 2023-11-25T04:43:34.098 INFO:teuthology.packaging:sha1: ec928368edcc5fc71884a56fd9fd821b148f4e30 2023-11-25T04:43:34.098 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2023-11-25T04:43:34.338 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7506.gec928368 2023-11-25T04:43:34.339 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-25T04:43:34.345 INFO:teuthology.task.internal:no buildpackages task found 2023-11-25T04:43:34.346 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-25T04:43:34.391 INFO:teuthology.task.internal:Saving configuration 2023-11-25T04:43:34.409 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-25T04:43:34.421 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-25T04:43:34.445 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-25 04:37:47.713884', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIQLzUj5DoDXok6ckR54zrIrE0YNr5/E8OHdWoHANIrAr9UBNH2fLFLMlj9n4BT4u5otCq4Gq1yMp/4oCqYCggY='} 2023-11-25T04:43:34.468 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-25 04:37:47.715842', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFtIGrbVuXwbnaMq4SVU04AhZvexOMXUoe+9Sjmpom43axHDTqNK2H6t0ELoI7LvcVNKRAUXDUS45FAfH/4JmEE='} 2023-11-25T04:43:34.468 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-25T04:43:34.479 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-11-25T04:43:34.479 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-11-25T04:43:34.479 INFO:teuthology.run_tasks:Running task console_log... 2023-11-25T04:43:34.645 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa50c932f70>, signals=[15]) 2023-11-25T04:43:34.646 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-25T04:43:34.654 INFO:teuthology.task.internal:Opening connections... 2023-11-25T04:43:34.654 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2023-11-25T04:43:34.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-25T04:43:34.762 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2023-11-25T04:43:34.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-25T04:43:34.850 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-25T04:43:34.861 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2023-11-25T04:43:34.880 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2023-11-25T04:43:34.880 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2023-11-25T04:43:34.938 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream" 2023-11-25T04:43:34.938 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="9" 2023-11-25T04:43:34.938 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2023-11-25T04:43:34.938 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2023-11-25T04:43:34.939 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="9" 2023-11-25T04:43:34.939 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el9" 2023-11-25T04:43:34.939 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-25T04:43:34.939 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2023-11-25T04:43:34.939 INFO:teuthology.orchestra.run.smithi007.stdout:LOGO="fedora-logo-icon" 2023-11-25T04:43:34.939 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-25T04:43:34.939 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/" 2023-11-25T04:43:34.939 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-25T04:43:34.939 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-25T04:43:34.939 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-25T04:43:34.940 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2023-11-25T04:43:34.984 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2023-11-25T04:43:35.002 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2023-11-25T04:43:35.002 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2023-11-25T04:43:35.059 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2023-11-25T04:43:35.060 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="9" 2023-11-25T04:43:35.060 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2023-11-25T04:43:35.060 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2023-11-25T04:43:35.060 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="9" 2023-11-25T04:43:35.060 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el9" 2023-11-25T04:43:35.060 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-25T04:43:35.060 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2023-11-25T04:43:35.060 INFO:teuthology.orchestra.run.smithi090.stdout:LOGO="fedora-logo-icon" 2023-11-25T04:43:35.060 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-25T04:43:35.060 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2023-11-25T04:43:35.060 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-25T04:43:35.060 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-25T04:43:35.060 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-25T04:43:35.061 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2023-11-25T04:43:35.085 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-25T04:43:35.101 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-25T04:43:35.121 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-25T04:43:35.121 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2023-11-25T04:43:35.124 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2023-11-25T04:43:35.147 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-25T04:43:35.155 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-25T04:43:35.155 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2023-11-25T04:43:35.182 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2023-11-25T04:43:35.214 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-25T04:43:35.276 INFO:teuthology.run_tasks:Running task kernel... 2023-11-25T04:43:35.292 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-25T04:43:35.293 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-25T04:43:35.293 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-25T04:43:35.294 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-25T04:43:35.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-25T04:43:35.309 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2023-11-25T04:43:35.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-25T04:43:35.311 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2023-11-25T04:43:35.366 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-361.el9.x86_64 2023-11-25T04:43:35.366 INFO:teuthology.task.kernel:Running kernel on smithi090: 5.14.0-361.el9.x86_64 2023-11-25T04:43:35.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum install -y kernel 2023-11-25T04:43:35.369 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-361.el9.x86_64 2023-11-25T04:43:35.369 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.14.0-361.el9.x86_64 2023-11-25T04:43:35.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2023-11-25T04:43:37.383 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 16 MB/s | 6.4 MB 00:00 2023-11-25T04:43:37.811 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-11-25T04:43:38.767 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2023-11-25T04:43:39.166 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2023-11-25T04:43:40.450 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 7.9 MB 00:01 2023-11-25T04:43:42.326 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 25 MB/s | 18 MB 00:00 2023-11-25T04:43:43.636 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 7.9 MB 00:04 2023-11-25T04:43:46.490 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 10 MB/s | 18 MB 00:01 2023-11-25T04:43:47.248 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 5.8 MB 00:01 2023-11-25T04:43:48.789 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 15 kB 00:00 2023-11-25T04:43:50.373 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.8 MB 00:00 2023-11-25T04:43:50.512 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2023-11-25T04:43:52.583 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 15 kB 00:01 2023-11-25T04:43:54.318 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2023-11-25T04:43:55.462 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2023-11-25T04:43:57.861 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-11-25T04:43:57.861 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-11-25T04:43:57.946 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:43:57.962 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:43:57.962 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2023-11-25T04:43:57.962 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:43:57.962 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:43:57.963 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 5.14.0-386.el9 baseos 5.5 M 2023-11-25T04:43:57.963 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-25T04:43:57.963 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core x86_64 5.14.0-386.el9 baseos 20 M 2023-11-25T04:43:57.963 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules x86_64 5.14.0-386.el9 baseos 38 M 2023-11-25T04:43:57.963 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core x86_64 5.14.0-386.el9 baseos 32 M 2023-11-25T04:43:57.963 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:43:57.964 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:43:57.964 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:43:57.964 INFO:teuthology.orchestra.run.smithi007.stdout:Install 4 Packages 2023-11-25T04:43:57.964 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:43:57.965 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 95 M 2023-11-25T04:43:57.965 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 121 M 2023-11-25T04:43:57.965 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:43:59.085 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2023-11-25T04:43:59.788 INFO:teuthology.orchestra.run.smithi007.stdout:(1/4): kernel-core-5.14.0-386.el9.x86_64.rpm 12 MB/s | 20 MB 00:01 2023-11-25T04:44:01.146 INFO:teuthology.orchestra.run.smithi007.stdout:(2/4): kernel-modules-core-5.14.0-386.el9.x86_6 24 MB/s | 32 MB 00:01 2023-11-25T04:44:01.475 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-11-25T04:44:01.475 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-11-25T04:44:01.588 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:44:01.593 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:44:01.594 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2023-11-25T04:44:01.594 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:44:01.594 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:44:01.594 INFO:teuthology.orchestra.run.smithi090.stdout: kernel x86_64 5.14.0-386.el9 baseos 5.5 M 2023-11-25T04:44:01.594 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2023-11-25T04:44:01.594 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core x86_64 5.14.0-386.el9 baseos 20 M 2023-11-25T04:44:01.594 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules x86_64 5.14.0-386.el9 baseos 38 M 2023-11-25T04:44:01.595 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core x86_64 5.14.0-386.el9 baseos 32 M 2023-11-25T04:44:01.595 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:44:01.595 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:44:01.595 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:44:01.595 INFO:teuthology.orchestra.run.smithi090.stdout:Install 4 Packages 2023-11-25T04:44:01.595 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:44:01.595 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 95 M 2023-11-25T04:44:01.596 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 121 M 2023-11-25T04:44:01.596 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:44:02.013 INFO:teuthology.orchestra.run.smithi007.stdout:(3/4): kernel-5.14.0-386.el9.x86_64.rpm 1.4 MB/s | 5.5 MB 00:03 2023-11-25T04:44:03.112 INFO:teuthology.orchestra.run.smithi090.stdout:(1/4): kernel-5.14.0-386.el9.x86_64.rpm 3.8 MB/s | 5.5 MB 00:01 2023-11-25T04:44:04.021 INFO:teuthology.orchestra.run.smithi007.stdout:(4/4): kernel-modules-5.14.0-386.el9.x86_64.rpm 6.4 MB/s | 38 MB 00:05 2023-11-25T04:44:04.021 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:44:04.022 INFO:teuthology.orchestra.run.smithi007.stdout:Total 16 MB/s | 95 MB 00:06 2023-11-25T04:44:04.911 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:44:05.262 INFO:teuthology.orchestra.run.smithi090.stdout:(2/4): kernel-modules-5.14.0-386.el9.x86_64.rpm 10 MB/s | 38 MB 00:03 2023-11-25T04:44:05.334 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:44:05.334 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:44:05.896 INFO:teuthology.orchestra.run.smithi090.stdout:(3/4): kernel-modules-core-5.14.0-386.el9.x86_6 12 MB/s | 32 MB 00:02 2023-11-25T04:44:06.738 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:44:06.738 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:44:07.646 INFO:teuthology.orchestra.run.smithi090.stdout:(4/4): kernel-core-5.14.0-386.el9.x86_64.rpm 3.4 MB/s | 20 MB 00:05 2023-11-25T04:44:07.646 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:44:07.647 INFO:teuthology.orchestra.run.smithi090.stdout:Total 16 MB/s | 95 MB 00:06 2023-11-25T04:44:08.500 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:44:08.867 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:44:08.867 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:44:09.509 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:44:10.142 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:44:10.143 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:44:10.188 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-core-5.14.0-386.el9.x86_64 1/4 2023-11-25T04:44:10.308 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-25T04:44:12.866 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:44:13.753 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-25T04:44:13.753 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-25T04:44:13.754 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-core-5.14.0-386.el9.x86_64 1/4 2023-11-25T04:44:13.754 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-25T04:44:16.452 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-25T04:44:16.648 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-25T04:44:18.342 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-25T04:44:18.461 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-5.14.0-386.el9.x86_64 4/4 2023-11-25T04:44:21.245 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-25T04:44:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-5.14.0-386.el9.x86_64 4/4 2023-11-25T04:44:23.442 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-core-5.14.0-386.el9.x86_64 4/4 2023-11-25T04:44:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-core-5.14.0-386.el9.x86_64 4/4 2023-11-25T04:44:54.254 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-386.el9.x86_64 4/4 2023-11-25T04:44:54.295 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-386.el9.x86_64 4/4 2023-11-25T04:44:54.754 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-5.14.0-386.el9.x86_64 4/4 2023-11-25T04:44:54.754 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-5.14.0-386.el9.x86_64 1/4 2023-11-25T04:44:54.754 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-25T04:44:54.754 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-25T04:44:55.189 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-core-5.14.0-386.el9.x86_64 4/4 2023-11-25T04:44:55.189 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:44:55.189 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:44:55.189 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-5.14.0-386.el9.x86_64 2023-11-25T04:44:55.189 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core-5.14.0-386.el9.x86_64 2023-11-25T04:44:55.189 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-5.14.0-386.el9.x86_64 2023-11-25T04:44:55.189 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core-5.14.0-386.el9.x86_64 2023-11-25T04:44:55.189 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:44:55.189 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:44:55.658 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2023-11-25T04:44:55.807 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-386.el9.x86_64 2023-11-25T04:44:55.808 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-386.el9.x86_64 2023-11-25T04:44:55.808 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-386.el9.x86_64 2023-11-25T04:44:55.808 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-25T04:44:55.808 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-25T04:44:55.808 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-11-25T04:44:55.808 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2023-11-25T04:44:55.808 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2023-11-25T04:44:55.933 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-386.el9.x86_64 2023-11-25T04:44:55.933 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-386.el9.x86_64 2023-11-25T04:44:55.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -qi grub2-tools 2023-11-25T04:44:55.977 INFO:teuthology.orchestra.run.smithi007.stdout:Name : grub2-tools 2023-11-25T04:44:55.978 INFO:teuthology.orchestra.run.smithi007.stdout:Epoch : 1 2023-11-25T04:44:55.978 INFO:teuthology.orchestra.run.smithi007.stdout:Version : 2.06 2023-11-25T04:44:55.978 INFO:teuthology.orchestra.run.smithi007.stdout:Release : 68.el9 2023-11-25T04:44:55.978 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: x86_64 2023-11-25T04:44:55.978 INFO:teuthology.orchestra.run.smithi007.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-25T04:44:55.978 INFO:teuthology.orchestra.run.smithi007.stdout:Group : Unspecified 2023-11-25T04:44:55.978 INFO:teuthology.orchestra.run.smithi007.stdout:Size : 8264254 2023-11-25T04:44:55.978 INFO:teuthology.orchestra.run.smithi007.stdout:License : GPLv3+ 2023-11-25T04:44:55.978 INFO:teuthology.orchestra.run.smithi007.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-25T04:44:55.978 INFO:teuthology.orchestra.run.smithi007.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-25T04:44:55.979 INFO:teuthology.orchestra.run.smithi007.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-25T04:44:55.979 INFO:teuthology.orchestra.run.smithi007.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-25T04:44:55.979 INFO:teuthology.orchestra.run.smithi007.stdout:Packager : builder@centos.org 2023-11-25T04:44:55.979 INFO:teuthology.orchestra.run.smithi007.stdout:Vendor : CentOS 2023-11-25T04:44:55.979 INFO:teuthology.orchestra.run.smithi007.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-25T04:44:55.979 INFO:teuthology.orchestra.run.smithi007.stdout:Summary : Support tools for GRUB. 2023-11-25T04:44:55.979 INFO:teuthology.orchestra.run.smithi007.stdout:Description : 2023-11-25T04:44:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:44:55.979 INFO:teuthology.orchestra.run.smithi007.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-25T04:44:55.979 INFO:teuthology.orchestra.run.smithi007.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-25T04:44:55.979 INFO:teuthology.orchestra.run.smithi007.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-25T04:44:55.979 INFO:teuthology.orchestra.run.smithi007.stdout:hardware devices. 2023-11-25T04:44:55.979 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:44:55.979 INFO:teuthology.orchestra.run.smithi007.stdout:This subpackage provides tools for support of all platforms. 2023-11-25T04:44:55.980 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-25T04:44:55.980 INFO:teuthology.task.kernel:Updating grub on smithi007 to boot 5.14.0-386.el9.x86_64 2023-11-25T04:44:55.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-25T04:44:56.351 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-386.el9.x86_64 4/4 2023-11-25T04:44:56.394 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-386.el9.x86_64 4/4 2023-11-25T04:44:56.602 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2023-11-25T04:44:56.788 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-5.14.0-386.el9.x86_64 4/4 2023-11-25T04:44:56.788 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-5.14.0-386.el9.x86_64 1/4 2023-11-25T04:44:56.789 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-25T04:44:56.789 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-25T04:44:57.219 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-core-5.14.0-386.el9.x86_64 4/4 2023-11-25T04:44:57.219 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:44:57.220 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:44:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-5.14.0-386.el9.x86_64 2023-11-25T04:44:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core-5.14.0-386.el9.x86_64 2023-11-25T04:44:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-5.14.0-386.el9.x86_64 2023-11-25T04:44:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core-5.14.0-386.el9.x86_64 2023-11-25T04:44:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:44:57.220 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:44:57.788 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2023-11-25T04:44:57.954 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-386.el9.x86_64 2023-11-25T04:44:57.954 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-386.el9.x86_64 2023-11-25T04:44:57.954 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-386.el9.x86_64 2023-11-25T04:44:57.954 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-25T04:44:57.954 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-25T04:44:57.954 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-11-25T04:44:57.954 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=None, version=distro) 2023-11-25T04:44:57.955 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2023-11-25T04:44:58.039 INFO:teuthology.orchestra.run.smithi007.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-25T04:44:58.073 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-386.el9.x86_64 2023-11-25T04:44:58.073 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-386.el9.x86_64 2023-11-25T04:44:58.073 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -qi grub2-tools 2023-11-25T04:44:58.086 INFO:teuthology.orchestra.run.smithi007.stderr:done 2023-11-25T04:44:58.089 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2023-11-25T04:44:58.104 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.R998GY3Isf 2023-11-25T04:44:58.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.R998GY3Isf 2023-11-25T04:44:58.112 INFO:teuthology.orchestra.run.smithi090.stdout:Name : grub2-tools 2023-11-25T04:44:58.113 INFO:teuthology.orchestra.run.smithi090.stdout:Epoch : 1 2023-11-25T04:44:58.113 INFO:teuthology.orchestra.run.smithi090.stdout:Version : 2.06 2023-11-25T04:44:58.113 INFO:teuthology.orchestra.run.smithi090.stdout:Release : 68.el9 2023-11-25T04:44:58.113 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: x86_64 2023-11-25T04:44:58.113 INFO:teuthology.orchestra.run.smithi090.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-25T04:44:58.113 INFO:teuthology.orchestra.run.smithi090.stdout:Group : Unspecified 2023-11-25T04:44:58.113 INFO:teuthology.orchestra.run.smithi090.stdout:Size : 8264254 2023-11-25T04:44:58.113 INFO:teuthology.orchestra.run.smithi090.stdout:License : GPLv3+ 2023-11-25T04:44:58.113 INFO:teuthology.orchestra.run.smithi090.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-25T04:44:58.113 INFO:teuthology.orchestra.run.smithi090.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-25T04:44:58.113 INFO:teuthology.orchestra.run.smithi090.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-25T04:44:58.113 INFO:teuthology.orchestra.run.smithi090.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-25T04:44:58.114 INFO:teuthology.orchestra.run.smithi090.stdout:Packager : builder@centos.org 2023-11-25T04:44:58.114 INFO:teuthology.orchestra.run.smithi090.stdout:Vendor : CentOS 2023-11-25T04:44:58.114 INFO:teuthology.orchestra.run.smithi090.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-25T04:44:58.114 INFO:teuthology.orchestra.run.smithi090.stdout:Summary : Support tools for GRUB. 2023-11-25T04:44:58.114 INFO:teuthology.orchestra.run.smithi090.stdout:Description : 2023-11-25T04:44:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:44:58.114 INFO:teuthology.orchestra.run.smithi090.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-25T04:44:58.114 INFO:teuthology.orchestra.run.smithi090.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-25T04:44:58.114 INFO:teuthology.orchestra.run.smithi090.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-25T04:44:58.114 INFO:teuthology.orchestra.run.smithi090.stdout:hardware devices. 2023-11-25T04:44:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:44:58.114 INFO:teuthology.orchestra.run.smithi090.stdout:This subpackage provides tools for support of all platforms. 2023-11-25T04:44:58.116 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-25T04:44:58.116 INFO:teuthology.task.kernel:Updating grub on smithi090 to boot 5.14.0-386.el9.x86_64 2023-11-25T04:44:58.116 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-25T04:44:58.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.R998GY3Isf 2023-11-25T04:44:58.332 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.R998GY3Isf is 6KB 2023-11-25T04:44:58.382 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.R998GY3Isf 2023-11-25T04:44:58.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo /bin/ls /boot/loader/entries || true 2023-11-25T04:44:58.469 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-25T04:44:58.469 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-25T04:44:58.469 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-25T04:44:58.469 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-386.el9.x86_64.conf 2023-11-25T04:44:58.470 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-386.el9.x86_64 2023-11-25T04:44:58.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2023-11-25T04:44:58.749 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2023-11-25T04:45:00.592 INFO:teuthology.orchestra.run.smithi090.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-25T04:45:00.613 INFO:teuthology.orchestra.run.smithi090.stderr:done 2023-11-25T04:45:00.615 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2023-11-25T04:45:00.646 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.c5DCmhmjqY 2023-11-25T04:45:00.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.c5DCmhmjqY 2023-11-25T04:45:00.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.c5DCmhmjqY 2023-11-25T04:45:00.873 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.c5DCmhmjqY is 6KB 2023-11-25T04:45:00.921 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.c5DCmhmjqY 2023-11-25T04:45:00.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo /bin/ls /boot/loader/entries || true 2023-11-25T04:45:01.002 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-25T04:45:01.006 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-25T04:45:01.006 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-25T04:45:01.006 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-386.el9.x86_64.conf 2023-11-25T04:45:01.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-386.el9.x86_64 2023-11-25T04:45:01.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2023-11-25T04:45:28.646 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-25T04:45:28.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-25T04:45:28.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-25T04:45:31.158 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-11-25T04:45:31.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-25T04:45:31.159 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-25T04:45:46.978 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2023-11-25T04:45:49.538 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2023-11-25T04:45:55.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-25T04:45:55.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-25T04:45:58.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-25T04:45:58.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-25T04:46:01.602 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2023-11-25T04:46:13.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-25T04:46:13.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-25T04:46:16.674 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2023-11-25T04:46:31.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-25T04:46:31.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-25T04:46:50.214 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2023-11-25T04:46:55.994 DEBUG:teuthology.orchestra.remote:timed out 2023-11-25T04:47:08.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-25T04:47:08.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-25T04:47:08.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-25T04:47:08.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-25T04:47:08.277 DEBUG:teuthology.orchestra.run.smithi007:> true 2023-11-25T04:47:08.564 DEBUG:teuthology.orchestra.run.smithi090:> true 2023-11-25T04:47:08.646 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-25T04:47:08.646 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-386.el9.x86_64"... 2023-11-25T04:47:08.647 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2023-11-25T04:47:08.663 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-386.el9.x86_64 2023-11-25T04:47:08.663 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-386.el9.x86_64 vs 5.14.0-386.el9.x86_64 2023-11-25T04:47:08.663 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-25T04:47:08.663 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-25T04:47:09.166 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-25T04:47:09.166 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-386.el9.x86_64"... 2023-11-25T04:47:09.166 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2023-11-25T04:47:09.180 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-386.el9.x86_64 2023-11-25T04:47:09.180 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-386.el9.x86_64 vs 5.14.0-386.el9.x86_64 2023-11-25T04:47:09.180 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-25T04:47:09.180 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-25T04:47:09.664 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-25T04:47:09.664 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-25T04:47:09.733 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2023-11-25T04:47:09.781 DEBUG:teuthology.parallel:result is None 2023-11-25T04:47:10.182 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-11-25T04:47:10.182 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-25T04:47:10.331 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2023-11-25T04:47:10.376 DEBUG:teuthology.parallel:result is None 2023-11-25T04:47:10.376 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-25T04:47:10.383 INFO:teuthology.task.internal:Creating test directory... 2023-11-25T04:47:10.383 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-25T04:47:10.386 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-25T04:47:10.404 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-25T04:47:10.485 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-25T04:47:10.527 INFO:teuthology.task.internal:Creating archive directory... 2023-11-25T04:47:10.528 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-25T04:47:10.531 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-25T04:47:10.560 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-25T04:47:10.578 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-25T04:47:10.579 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-25T04:47:10.595 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-25T04:47:10.628 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-25T04:47:10.638 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-25T04:47:10.641 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-25T04:47:10.657 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-25T04:47:10.658 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-25T04:47:10.665 INFO:teuthology.task.internal:Configuring sudo... 2023-11-25T04:47:10.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-25T04:47:10.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-25T04:47:10.710 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-25T04:47:10.720 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-25T04:47:10.721 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-25T04:47:10.743 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-25T04:47:10.767 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-25T04:47:10.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-25T04:47:10.910 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-25T04:47:10.953 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-25T04:47:11.002 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T04:47:11.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-25T04:47:11.074 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-25T04:47:11.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-25T04:47:11.168 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-25T04:47:11.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-25T04:47:11.293 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T04:47:11.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-25T04:47:11.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2023-11-25T04:47:11.363 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2023-11-25T04:47:11.411 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-25T04:47:11.536 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-25T04:47:11.901 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-25T04:47:11.912 INFO:teuthology.task.internal:Starting timer... 2023-11-25T04:47:11.912 INFO:teuthology.run_tasks:Running task pcp... 2023-11-25T04:47:11.950 INFO:teuthology.run_tasks:Running task selinux... 2023-11-25T04:47:11.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2023-11-25T04:47:12.057 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2023-11-25T04:47:12.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2023-11-25T04:47:12.168 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2023-11-25T04:47:12.171 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-25T04:47:12.171 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2023-11-25T04:47:12.193 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2023-11-25T04:47:12.193 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2023-11-25T04:47:12.225 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2023-11-25T04:47:12.225 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi090.front.sepia.ceph.com': 'permissive'} 2023-11-25T04:47:12.225 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-25T04:47:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-25T04:47:12.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-25T04:47:12.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-25T04:47:12.325 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-25T04:47:12.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2023-11-25T04:47:12.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2023-11-25T04:47:12.418 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-25T04:47:12.429 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-25T04:47:12.429 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-25T04:47:12.477 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-25T04:47:12.479 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi090.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-25T04:53:08.057 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi090.front.sepia.ceph.com')] 2023-11-25T04:53:08.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-25T04:53:08.059 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-25T04:53:08.157 DEBUG:teuthology.orchestra.run.smithi007:> true 2023-11-25T04:53:08.229 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-25T04:53:08.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-25T04:53:08.230 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-25T04:53:08.315 DEBUG:teuthology.orchestra.run.smithi090:> true 2023-11-25T04:53:08.390 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-25T04:53:08.390 INFO:teuthology.run_tasks:Running task clock... 2023-11-25T04:53:08.406 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-25T04:53:08.406 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-25T04:53:08.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-25T04:53:08.409 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-25T04:53:08.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-25T04:53:08.443 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-25T04:53:08.462 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-25T04:53:08.463 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-25T04:53:08.479 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-25T04:53:08.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2023-11-25T04:53:08.508 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2023-11-25T04:53:08.516 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2023-11-25T04:53:08.522 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2023-11-25T04:53:08.536 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-25T04:53:08.539 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-25T04:53:08.555 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-25T04:53:08.556 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-25T04:53:08.598 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2023-11-25T04:53:08.601 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-25T04:53:08.601 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2023-11-25T04:53:08.601 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-25T04:53:08.602 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-25T04:53:08.602 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-25T04:53:08.602 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-25T04:53:08.603 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2023-11-25T04:53:08.606 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-25T04:53:08.606 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2023-11-25T04:53:08.606 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-25T04:53:08.606 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-25T04:53:08.607 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-25T04:53:08.607 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-25T04:53:08.609 INFO:teuthology.run_tasks:Running task install... 2023-11-25T04:53:08.618 DEBUG:teuthology.task.install:project ceph 2023-11-25T04:53:08.618 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'ec928368edcc5fc71884a56fd9fd821b148f4e30'}} 2023-11-25T04:53:08.618 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'ec928368edcc5fc71884a56fd9fd821b148f4e30'} 2023-11-25T04:53:08.618 INFO:teuthology.task.install:Using flavor: default 2023-11-25T04:53:08.624 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-11-25T04:53:08.624 INFO:teuthology.task.install:extra packages: [] 2023-11-25T04:53:08.624 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ec928368edcc5fc71884a56fd9fd821b148f4e30', 'tag': None, 'wait_for_package': False} 2023-11-25T04:53:08.625 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ec928368edcc5fc71884a56fd9fd821b148f4e30 2023-11-25T04:53:08.627 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ec928368edcc5fc71884a56fd9fd821b148f4e30', 'tag': None, 'wait_for_package': False} 2023-11-25T04:53:08.627 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ec928368edcc5fc71884a56fd9fd821b148f4e30 2023-11-25T04:53:08.788 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/ec928368edcc5fc71884a56fd9fd821b148f4e30/centos/9/flavors/default/ 2023-11-25T04:53:08.788 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7506.gec928368 2023-11-25T04:53:08.793 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/ec928368edcc5fc71884a56fd9fd821b148f4e30/centos/9/flavors/default/ 2023-11-25T04:53:08.793 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7506.gec928368 2023-11-25T04:53:08.963 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/ec928368edcc5fc71884a56fd9fd821b148f4e30/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/ec928368edcc5fc71884a56fd9fd821b148f4e30/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/ec928368edcc5fc71884a56fd9fd821b148f4e30/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-25T04:53:08.964 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T04:53:08.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-25T04:53:08.966 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/ec928368edcc5fc71884a56fd9fd821b148f4e30/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/ec928368edcc5fc71884a56fd9fd821b148f4e30/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/ec928368edcc5fc71884a56fd9fd821b148f4e30/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-25T04:53:08.966 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T04:53:08.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-25T04:53:08.994 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-11-25T04:53:08.995 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ec928368edcc5fc71884a56fd9fd821b148f4e30/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-25T04:53:08.998 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-11-25T04:53:08.999 DEBUG:teuthology.orchestra.run.smithi007:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ec928368edcc5fc71884a56fd9fd821b148f4e30/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-25T04:53:09.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-25T04:53:09.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-25T04:53:09.143 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-25T04:53:09.157 DEBUG:teuthology.orchestra.run.smithi007:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-25T04:53:09.187 INFO:teuthology.orchestra.run.smithi007.stdout:check_obsoletes = 1 2023-11-25T04:53:09.188 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum clean all 2023-11-25T04:53:09.210 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2023-11-25T04:53:09.211 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2023-11-25T04:53:09.438 INFO:teuthology.orchestra.run.smithi007.stdout:73 files removed 2023-11-25T04:53:09.458 INFO:teuthology.orchestra.run.smithi090.stdout:73 files removed 2023-11-25T04:53:09.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw 2023-11-25T04:53:09.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw 2023-11-25T04:53:10.265 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 168 kB/s | 78 kB 00:00 2023-11-25T04:53:10.294 INFO:teuthology.orchestra.run.smithi007.stdout:ceph packages for x86_64 158 kB/s | 78 kB 00:00 2023-11-25T04:53:10.657 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 28 kB/s | 9.5 kB 00:00 2023-11-25T04:53:10.678 INFO:teuthology.orchestra.run.smithi007.stdout:ceph noarch packages 28 kB/s | 9.5 kB 00:00 2023-11-25T04:53:11.175 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2023-11-25T04:53:11.229 INFO:teuthology.orchestra.run.smithi007.stdout:ceph source packages 3.7 kB/s | 1.9 kB 00:00 2023-11-25T04:53:11.760 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 12 MB/s | 6.4 MB 00:00 2023-11-25T04:53:11.898 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 10 MB/s | 6.4 MB 00:00 2023-11-25T04:53:13.186 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2023-11-25T04:53:13.357 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2023-11-25T04:53:14.944 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 7.9 MB 00:01 2023-11-25T04:53:15.339 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 7.9 MB 00:01 2023-11-25T04:53:16.779 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 25 MB/s | 18 MB 00:00 2023-11-25T04:53:17.240 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 25 MB/s | 18 MB 00:00 2023-11-25T04:53:21.023 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 5.8 MB 00:00 2023-11-25T04:53:21.874 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 5.8 MB 00:01 2023-11-25T04:53:23.333 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 15 kB 00:00 2023-11-25T04:53:23.641 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 10 kB/s | 15 kB 00:01 2023-11-25T04:53:24.509 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 27 MB/s | 20 MB 00:00 2023-11-25T04:53:25.561 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 9.4 MB/s | 20 MB 00:02 2023-11-25T04:53:29.210 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2023-11-25T04:53:30.630 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-11-25T04:53:31.893 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:53:31.911 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:53:31.911 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2023-11-25T04:53:31.911 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:53:31.911 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:53:31.911 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:18.0.0-7506.gec928368.el9 ceph 8.7 M 2023-11-25T04:53:31.911 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2023-11-25T04:53:31.911 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:18.0.0-7506.gec928368.el9 ceph 3.3 M 2023-11-25T04:53:31.911 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:18.0.0-7506.gec928368.el9 ceph 3.0 M 2023-11-25T04:53:31.911 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2023-11-25T04:53:31.911 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options 2023-11-25T04:53:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-25T04:53:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:18.0.0-7506.gec928368.el9 ceph 5.2 M 2023-11-25T04:53:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:18.0.0-7506.gec928368.el9 ceph 19 M 2023-11-25T04:53:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:18.0.0-7506.gec928368.el9 ceph 24 k 2023-11-25T04:53:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-25T04:53:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-25T04:53:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-25T04:53:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:18.0.0-7506.gec928368.el9 ceph 719 k 2023-11-25T04:53:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-25T04:53:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-25T04:53:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-11-25T04:53:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:18.0.0-7506.gec928368.el9 ceph 476 k 2023-11-25T04:53:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-25T04:53:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:18.0.0-7506.gec928368.el9 ceph 4.8 M 2023-11-25T04:53:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-25T04:53:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-25T04:53:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-25T04:53:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse 2023-11-25T04:53:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: x86_64 2:18.0.0-7506.gec928368.el9 ceph 45 k 2023-11-25T04:53:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:18.0.0-7506.gec928368.el9 ceph 129 k 2023-11-25T04:53:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:18.0.0-7506.gec928368.el9 ceph 162 k 2023-11-25T04:53:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-25T04:53:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:18.0.0-7506.gec928368.el9 ceph 320 k 2023-11-25T04:53:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:18.0.0-7506.gec928368.el9 ceph 298 k 2023-11-25T04:53:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:18.0.0-7506.gec928368.el9 ceph 99 k 2023-11-25T04:53:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-25T04:53:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-25T04:53:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:53:31.914 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:53:31.914 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:53:31.914 INFO:teuthology.orchestra.run.smithi090.stdout:Install 27 Packages 2023-11-25T04:53:31.914 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 2 Packages 2023-11-25T04:53:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:53:31.915 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 55 M 2023-11-25T04:53:31.915 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:53:32.487 INFO:teuthology.orchestra.run.smithi090.stdout:(1/29): ceph-base-18.0.0-7506.gec928368.el9.x86 10 MB/s | 5.2 MB 00:00 2023-11-25T04:53:32.529 INFO:teuthology.orchestra.run.smithi090.stdout:(2/29): ceph-selinux-18.0.0-7506.gec928368.el9. 576 kB/s | 24 kB 00:00 2023-11-25T04:53:32.604 INFO:teuthology.orchestra.run.smithi090.stdout:(3/29): libcephfs2-18.0.0-7506.gec928368.el9.x8 9.4 MB/s | 719 kB 00:00 2023-11-25T04:53:32.671 INFO:teuthology.orchestra.run.smithi090.stdout:(4/29): libradosstriper1-18.0.0-7506.gec928368. 7.0 MB/s | 476 kB 00:00 2023-11-25T04:53:32.998 INFO:teuthology.orchestra.run.smithi090.stdout:(5/29): librgw2-18.0.0-7506.gec928368.el9.x86_6 15 MB/s | 4.8 MB 00:00 2023-11-25T04:53:33.048 INFO:teuthology.orchestra.run.smithi090.stdout:(6/29): python3-ceph-argparse-18.0.0-7506.gec92 903 kB/s | 45 kB 00:00 2023-11-25T04:53:33.106 INFO:teuthology.orchestra.run.smithi090.stdout:(7/29): python3-ceph-common-18.0.0-7506.gec9283 2.2 MB/s | 129 kB 00:00 2023-11-25T04:53:33.173 INFO:teuthology.orchestra.run.smithi090.stdout:(8/29): python3-cephfs-18.0.0-7506.gec928368.el 2.4 MB/s | 162 kB 00:00 2023-11-25T04:53:33.240 INFO:teuthology.orchestra.run.smithi090.stdout:(9/29): python3-rados-18.0.0-7506.gec928368.el9 4.7 MB/s | 320 kB 00:00 2023-11-25T04:53:33.316 INFO:teuthology.orchestra.run.smithi090.stdout:(10/29): python3-rbd-18.0.0-7506.gec928368.el9. 3.9 MB/s | 298 kB 00:00 2023-11-25T04:53:33.375 INFO:teuthology.orchestra.run.smithi090.stdout:(11/29): python3-rgw-18.0.0-7506.gec928368.el9. 1.7 MB/s | 99 kB 00:00 2023-11-25T04:53:33.437 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:53:33.449 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:53:33.450 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-25T04:53:33.450 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:53:33.450 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:53:33.450 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw x86_64 2:18.0.0-7506.gec928368.el9 ceph 8.7 M 2023-11-25T04:53:33.450 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrading: 2023-11-25T04:53:33.450 INFO:teuthology.orchestra.run.smithi007.stdout: librados2 x86_64 2:18.0.0-7506.gec928368.el9 ceph 3.3 M 2023-11-25T04:53:33.450 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1 x86_64 2:18.0.0-7506.gec928368.el9 ceph 3.0 M 2023-11-25T04:53:33.450 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-25T04:53:33.450 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options 2023-11-25T04:53:33.450 INFO:teuthology.orchestra.run.smithi007.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-25T04:53:33.450 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base x86_64 2:18.0.0-7506.gec928368.el9 ceph 5.2 M 2023-11-25T04:53:33.451 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common x86_64 2:18.0.0-7506.gec928368.el9 ceph 19 M 2023-11-25T04:53:33.451 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux x86_64 2:18.0.0-7506.gec928368.el9 ceph 24 k 2023-11-25T04:53:33.451 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-25T04:53:33.451 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-25T04:53:33.451 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-25T04:53:33.451 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2 x86_64 2:18.0.0-7506.gec928368.el9 ceph 719 k 2023-11-25T04:53:33.451 INFO:teuthology.orchestra.run.smithi007.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-25T04:53:33.451 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-25T04:53:33.451 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-11-25T04:53:33.451 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 x86_64 2:18.0.0-7506.gec928368.el9 ceph 476 k 2023-11-25T04:53:33.451 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-25T04:53:33.452 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2 x86_64 2:18.0.0-7506.gec928368.el9 ceph 4.8 M 2023-11-25T04:53:33.452 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-25T04:53:33.452 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-25T04:53:33.452 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-25T04:53:33.452 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse 2023-11-25T04:53:33.452 INFO:teuthology.orchestra.run.smithi007.stdout: x86_64 2:18.0.0-7506.gec928368.el9 ceph 45 k 2023-11-25T04:53:33.452 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common x86_64 2:18.0.0-7506.gec928368.el9 ceph 129 k 2023-11-25T04:53:33.452 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs x86_64 2:18.0.0-7506.gec928368.el9 ceph 162 k 2023-11-25T04:53:33.452 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-25T04:53:33.452 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados x86_64 2:18.0.0-7506.gec928368.el9 ceph 320 k 2023-11-25T04:53:33.452 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd x86_64 2:18.0.0-7506.gec928368.el9 ceph 298 k 2023-11-25T04:53:33.452 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw x86_64 2:18.0.0-7506.gec928368.el9 ceph 99 k 2023-11-25T04:53:33.452 INFO:teuthology.orchestra.run.smithi007.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-25T04:53:33.453 INFO:teuthology.orchestra.run.smithi007.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-25T04:53:33.453 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:53:33.453 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:53:33.453 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:53:33.453 INFO:teuthology.orchestra.run.smithi007.stdout:Install 27 Packages 2023-11-25T04:53:33.453 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrade 2 Packages 2023-11-25T04:53:33.453 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:53:33.453 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 55 M 2023-11-25T04:53:33.454 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:53:33.483 INFO:teuthology.orchestra.run.smithi090.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 979 kB/s | 106 kB 00:00 2023-11-25T04:53:33.534 INFO:teuthology.orchestra.run.smithi090.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2023-11-25T04:53:33.584 INFO:teuthology.orchestra.run.smithi090.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 872 kB/s | 44 kB 00:00 2023-11-25T04:53:33.652 INFO:teuthology.orchestra.run.smithi090.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2023-11-25T04:53:33.702 INFO:teuthology.orchestra.run.smithi090.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2023-11-25T04:53:33.744 INFO:teuthology.orchestra.run.smithi090.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 983 kB/s | 41 kB 00:00 2023-11-25T04:53:34.011 INFO:teuthology.orchestra.run.smithi090.stdout:(18/29): ceph-radosgw-18.0.0-7506.gec928368.el9 4.2 MB/s | 8.7 MB 00:02 2023-11-25T04:53:34.062 INFO:teuthology.orchestra.run.smithi090.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 973 kB/s | 309 kB 00:00 2023-11-25T04:53:34.362 INFO:teuthology.orchestra.run.smithi090.stdout:(20/29): ceph-common-18.0.0-7506.gec928368.el9. 7.9 MB/s | 19 MB 00:02 2023-11-25T04:53:34.446 INFO:teuthology.orchestra.run.smithi090.stdout:(21/29): libarrow-9.0.0-9.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2023-11-25T04:53:34.471 INFO:teuthology.orchestra.run.smithi090.stdout:(22/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 60 kB/s | 24 kB 00:00 2023-11-25T04:53:34.496 INFO:teuthology.orchestra.run.smithi090.stdout:(23/29): liboath-2.6.7-2.el9.x86_64.rpm 370 kB/s | 49 kB 00:00 2023-11-25T04:53:34.522 INFO:teuthology.orchestra.run.smithi090.stdout:(24/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2023-11-25T04:53:34.527 INFO:teuthology.orchestra.run.smithi007.stdout:(1/29): ceph-radosgw-18.0.0-7506.gec928368.el9. 8.5 MB/s | 8.7 MB 00:01 2023-11-25T04:53:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:(25/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 8.9 MB/s | 837 kB 00:00 2023-11-25T04:53:34.585 INFO:teuthology.orchestra.run.smithi007.stdout:(2/29): ceph-selinux-18.0.0-7506.gec928368.el9. 415 kB/s | 24 kB 00:00 2023-11-25T04:53:34.589 INFO:teuthology.orchestra.run.smithi090.stdout:(26/29): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2023-11-25T04:53:34.631 INFO:teuthology.orchestra.run.smithi090.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2023-11-25T04:53:34.669 INFO:teuthology.orchestra.run.smithi007.stdout:(3/29): libcephfs2-18.0.0-7506.gec928368.el9.x8 8.4 MB/s | 719 kB 00:00 2023-11-25T04:53:34.886 INFO:teuthology.orchestra.run.smithi007.stdout:(4/29): libradosstriper1-18.0.0-7506.gec928368. 2.1 MB/s | 476 kB 00:00 2023-11-25T04:53:34.906 INFO:teuthology.orchestra.run.smithi090.stdout:(28/29): librbd1-18.0.0-7506.gec928368.el9.x86_ 9.6 MB/s | 3.0 MB 00:00 2023-11-25T04:53:35.040 INFO:teuthology.orchestra.run.smithi090.stdout:(29/29): librados2-18.0.0-7506.gec928368.el9.x8 7.4 MB/s | 3.3 MB 00:00 2023-11-25T04:53:35.044 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:53:35.044 INFO:teuthology.orchestra.run.smithi090.stdout:Total 18 MB/s | 55 MB 00:03 2023-11-25T04:53:35.045 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:53:35.080 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:53:35.080 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:53:35.161 INFO:teuthology.orchestra.run.smithi007.stdout:(5/29): ceph-base-18.0.0-7506.gec928368.el9.x86 3.1 MB/s | 5.2 MB 00:01 2023-11-25T04:53:35.203 INFO:teuthology.orchestra.run.smithi007.stdout:(6/29): python3-ceph-argparse-18.0.0-7506.gec92 1.1 MB/s | 45 kB 00:00 2023-11-25T04:53:35.270 INFO:teuthology.orchestra.run.smithi007.stdout:(7/29): librgw2-18.0.0-7506.gec928368.el9.x86_6 12 MB/s | 4.8 MB 00:00 2023-11-25T04:53:35.295 INFO:teuthology.orchestra.run.smithi007.stdout:(8/29): python3-ceph-common-18.0.0-7506.gec9283 1.4 MB/s | 129 kB 00:00 2023-11-25T04:53:35.362 INFO:teuthology.orchestra.run.smithi007.stdout:(9/29): python3-cephfs-18.0.0-7506.gec928368.el 1.7 MB/s | 162 kB 00:00 2023-11-25T04:53:35.388 INFO:teuthology.orchestra.run.smithi007.stdout:(10/29): python3-rados-18.0.0-7506.gec928368.el 3.4 MB/s | 320 kB 00:00 2023-11-25T04:53:35.438 INFO:teuthology.orchestra.run.smithi007.stdout:(11/29): python3-rbd-18.0.0-7506.gec928368.el9. 3.9 MB/s | 298 kB 00:00 2023-11-25T04:53:35.463 INFO:teuthology.orchestra.run.smithi007.stdout:(12/29): python3-rgw-18.0.0-7506.gec928368.el9. 1.3 MB/s | 99 kB 00:00 2023-11-25T04:53:35.533 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:53:35.534 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:53:35.647 INFO:teuthology.orchestra.run.smithi007.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 507 kB/s | 106 kB 00:00 2023-11-25T04:53:35.680 INFO:teuthology.orchestra.run.smithi007.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 831 kB/s | 159 kB 00:00 2023-11-25T04:53:35.697 INFO:teuthology.orchestra.run.smithi007.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 879 kB/s | 44 kB 00:00 2023-11-25T04:53:35.764 INFO:teuthology.orchestra.run.smithi007.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2023-11-25T04:53:35.789 INFO:teuthology.orchestra.run.smithi007.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2023-11-25T04:53:35.806 INFO:teuthology.orchestra.run.smithi007.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 41 kB 00:00 2023-11-25T04:53:35.948 INFO:teuthology.orchestra.run.smithi007.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.9 MB/s | 309 kB 00:00 2023-11-25T04:53:35.982 INFO:teuthology.orchestra.run.smithi007.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 733 kB/s | 24 kB 00:00 2023-11-25T04:53:36.015 INFO:teuthology.orchestra.run.smithi007.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2023-11-25T04:53:36.066 INFO:teuthology.orchestra.run.smithi007.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2023-11-25T04:53:36.174 INFO:teuthology.orchestra.run.smithi007.stdout:(23/29): libarrow-9.0.0-9.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2023-11-25T04:53:36.216 INFO:teuthology.orchestra.run.smithi007.stdout:(24/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 5.4 MB/s | 837 kB 00:00 2023-11-25T04:53:36.242 INFO:teuthology.orchestra.run.smithi007.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2023-11-25T04:53:36.359 INFO:teuthology.orchestra.run.smithi007.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2023-11-25T04:53:36.826 INFO:teuthology.orchestra.run.smithi007.stdout:(27/29): ceph-common-18.0.0-7506.gec928368.el9. 5.7 MB/s | 19 MB 00:03 2023-11-25T04:53:36.848 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:53:36.884 INFO:teuthology.orchestra.run.smithi007.stdout:(28/29): librados2-18.0.0-7506.gec928368.el9.x8 5.2 MB/s | 3.3 MB 00:00 2023-11-25T04:53:36.942 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-11-25T04:53:36.943 INFO:teuthology.orchestra.run.smithi007.stdout:(29/29): librbd1-18.0.0-7506.gec928368.el9.x86_ 5.2 MB/s | 3.0 MB 00:00 2023-11-25T04:53:36.947 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:53:36.947 INFO:teuthology.orchestra.run.smithi007.stdout:Total 16 MB/s | 55 MB 00:03 2023-11-25T04:53:36.948 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:53:36.979 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-11-25T04:53:36.983 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:53:36.984 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:53:37.046 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-11-25T04:53:37.411 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-11-25T04:53:37.456 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:53:37.456 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:53:37.462 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:18.0.0-7506.gec928368.el9.x86_64 5/31 2023-11-25T04:53:37.597 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:18.0.0-7506.gec928368.el9.x86_64 5/31 2023-11-25T04:53:37.689 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:18.0.0-7506.gec928368.el9.x86_64 6/31 2023-11-25T04:53:37.714 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:18.0.0-7506.gec928368.el9.x86_64 7/31 2023-11-25T04:53:37.760 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:18.0.0-7506.gec928368.el9.x86_64 7/31 2023-11-25T04:53:37.789 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-11-25T04:53:37.833 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:18.0.0-7506.gec928368.el9. 9/31 2023-11-25T04:53:37.903 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:18.0.0-7506.gec928368.el9.x86_64 10/31 2023-11-25T04:53:37.934 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:18.0.0-7506.gec928368.el9.x86_6 11/31 2023-11-25T04:53:37.999 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7506.gec928368.el9.x86_6 11/31 2023-11-25T04:53:38.105 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-11-25T04:53:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-11-25T04:53:38.183 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-11-25T04:53:38.216 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-11-25T04:53:38.254 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-11-25T04:53:38.287 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-11-25T04:53:38.591 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-11-25T04:53:38.617 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:18.0.0-7506.gec928368.el9.x86_64 19/31 2023-11-25T04:53:38.707 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:18.0.0-7506.gec928368.el9.x86_64 19/31 2023-11-25T04:53:38.739 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:18.0.0-7506.gec928368.el9.x86_64 20/31 2023-11-25T04:53:38.775 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-11-25T04:53:38.870 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:53:38.940 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-11-25T04:53:38.975 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-11-25T04:53:39.026 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-11-25T04:53:39.203 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-11-25T04:53:39.238 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:18.0.0-7506.gec928368.el9.x86_64 23/31 2023-11-25T04:53:39.310 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:18.0.0-7506.gec928368.el9.x86_64 23/31 2023-11-25T04:53:39.385 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:18.0.0-7506.gec928368.el9.x86_64 24/31 2023-11-25T04:53:39.416 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:18.0.0-7506.gec928368.el9.x8 25/31 2023-11-25T04:53:39.425 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-11-25T04:53:39.466 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librados2-2:18.0.0-7506.gec928368.el9.x86_64 5/31 2023-11-25T04:53:39.568 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:18.0.0-7506.gec928368.el9.x86_64 5/31 2023-11-25T04:53:39.654 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-2:18.0.0-7506.gec928368.el9.x86_64 6/31 2023-11-25T04:53:39.684 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-2:18.0.0-7506.gec928368.el9.x86_64 7/31 2023-11-25T04:53:39.729 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-2:18.0.0-7506.gec928368.el9.x86_64 7/31 2023-11-25T04:53:39.759 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-11-25T04:53:39.811 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-argparse-2:18.0.0-7506.gec928368.el9. 9/31 2023-11-25T04:53:39.879 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-2:18.0.0-7506.gec928368.el9.x86_64 10/31 2023-11-25T04:53:39.903 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradosstriper1-2:18.0.0-7506.gec928368.el9.x86_6 11/31 2023-11-25T04:53:39.960 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7506.gec928368.el9.x86_6 11/31 2023-11-25T04:53:40.070 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-11-25T04:53:40.102 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-11-25T04:53:40.143 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-11-25T04:53:40.175 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-11-25T04:53:40.204 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-11-25T04:53:40.237 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-11-25T04:53:40.560 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-11-25T04:53:40.591 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librbd1-2:18.0.0-7506.gec928368.el9.x86_64 19/31 2023-11-25T04:53:40.683 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:18.0.0-7506.gec928368.el9.x86_64 19/31 2023-11-25T04:53:40.713 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-2:18.0.0-7506.gec928368.el9.x86_64 20/31 2023-11-25T04:53:40.749 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-11-25T04:53:41.200 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-11-25T04:53:41.227 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librgw2-2:18.0.0-7506.gec928368.el9.x86_64 23/31 2023-11-25T04:53:41.285 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librgw2-2:18.0.0-7506.gec928368.el9.x86_64 23/31 2023-11-25T04:53:41.372 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-2:18.0.0-7506.gec928368.el9.x86_64 24/31 2023-11-25T04:53:41.400 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-common-2:18.0.0-7506.gec928368.el9.x8 25/31 2023-11-25T04:53:41.629 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:18.0.0-7506.gec928368.el9.x86_64 26/31 2023-11-25T04:53:41.665 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:18.0.0-7506.gec928368.el9.x86_64 26/31 2023-11-25T04:53:42.226 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:18.0.0-7506.gec928368.el9.x86_64 26/31 2023-11-25T04:53:42.270 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:18.0.0-7506.gec928368.el9.x86_64 27/31 2023-11-25T04:53:42.354 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:18.0.0-7506.gec928368.el9.x86_64 27/31 2023-11-25T04:53:42.355 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-25T04:53:42.355 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-25T04:53:42.355 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:53:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:18.0.0-7506.gec928368.el9.x86_64 28/31 2023-11-25T04:53:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:18.0.0-7506.gec928368.el9.x86_64 26/31 2023-11-25T04:53:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-2:18.0.0-7506.gec928368.el9.x86_64 26/31 2023-11-25T04:53:44.225 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:18.0.0-7506.gec928368.el9.x86_64 26/31 2023-11-25T04:53:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-2:18.0.0-7506.gec928368.el9.x86_64 27/31 2023-11-25T04:53:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-2:18.0.0-7506.gec928368.el9.x86_64 27/31 2023-11-25T04:53:44.393 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-25T04:53:44.393 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-25T04:53:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:53:44.434 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-selinux-2:18.0.0-7506.gec928368.el9.x86_64 28/31 2023-11-25T04:53:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7506.gec928368.el9.x86_64 28/31 2023-11-25T04:53:52.456 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2023-11-25T04:53:52.456 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2023-11-25T04:53:52.456 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2023-11-25T04:53:52.456 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2023-11-25T04:53:52.456 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2023-11-25T04:53:52.456 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /root 2023-11-25T04:53:52.456 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2023-11-25T04:53:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:53:53.323 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:18.0.0-7506.gec928368.el9.x86_64 29/31 2023-11-25T04:53:53.348 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7506.gec928368.el9.x86_64 29/31 2023-11-25T04:53:53.349 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:53:53.350 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-25T04:53:53.350 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-25T04:53:53.351 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-25T04:53:53.351 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:53:53.354 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-25T04:53:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-25T04:53:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-25T04:53:54.533 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7506.gec928368.el9.x86_64 28/31 2023-11-25T04:53:54.534 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /sys 2023-11-25T04:53:54.534 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /proc 2023-11-25T04:53:54.534 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /mnt 2023-11-25T04:53:54.534 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /var/tmp 2023-11-25T04:53:54.534 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /home 2023-11-25T04:53:54.534 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /root 2023-11-25T04:53:54.534 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /tmp 2023-11-25T04:53:54.534 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:53:54.773 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-25T04:53:54.773 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:18.0.0-7506.gec928368.el9.x86_64 1/31 2023-11-25T04:53:54.773 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:18.0.0-7506.gec928368.el9.x86_64 2/31 2023-11-25T04:53:54.773 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:18.0.0-7506.gec928368.el9.x86_64 3/31 2023-11-25T04:53:54.773 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:18.0.0-7506.gec928368.el9.x86_64 4/31 2023-11-25T04:53:54.773 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:18.0.0-7506.gec928368.el9.x86_64 5/31 2023-11-25T04:53:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:18.0.0-7506.gec928368.el9.x86_6 6/31 2023-11-25T04:53:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:18.0.0-7506.gec928368.el9.x86_64 7/31 2023-11-25T04:53:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7506.gec928368.el9. 8/31 2023-11-25T04:53:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:18.0.0-7506.gec928368.el9.x8 9/31 2023-11-25T04:53:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:18.0.0-7506.gec928368.el9.x86_64 10/31 2023-11-25T04:53:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:18.0.0-7506.gec928368.el9.x86_64 11/31 2023-11-25T04:53:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:18.0.0-7506.gec928368.el9.x86_64 12/31 2023-11-25T04:53:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:18.0.0-7506.gec928368.el9.x86_64 13/31 2023-11-25T04:53:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-11-25T04:53:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-11-25T04:53:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-11-25T04:53:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-11-25T04:53:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-11-25T04:53:54.775 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-11-25T04:53:54.775 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-11-25T04:53:54.775 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-11-25T04:53:54.775 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-11-25T04:53:54.775 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-11-25T04:53:54.775 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-11-25T04:53:54.775 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-11-25T04:53:54.775 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-11-25T04:53:54.777 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-11-25T04:53:54.777 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:18.0.0-7506.gec928368.el9.x86_64 28/31 2023-11-25T04:53:54.777 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-11-25T04:53:54.777 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:18.0.0-7506.gec928368.el9.x86_64 30/31 2023-11-25T04:53:55.401 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-2:18.0.0-7506.gec928368.el9.x86_64 29/31 2023-11-25T04:53:55.427 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7506.gec928368.el9.x86_64 29/31 2023-11-25T04:53:55.428 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:53:55.428 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-25T04:53:55.428 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-25T04:53:55.428 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-25T04:53:55.428 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:53:55.430 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-25T04:53:55.504 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-25T04:53:55.504 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-25T04:53:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-11-25T04:53:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:53:55.714 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2023-11-25T04:53:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:55.714 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:53:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-25T04:53:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-25T04:53:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-25T04:53:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-25T04:53:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-25T04:53:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-25T04:53:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-25T04:53:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-25T04:53:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-25T04:53:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-25T04:53:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-25T04:53:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-25T04:53:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: re2-1:20211101-3.el9.x86_64 2023-11-25T04:53:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-25T04:53:55.717 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:53:55.717 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:53:56.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-test 2023-11-25T04:53:56.670 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-25T04:53:56.670 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-2:18.0.0-7506.gec928368.el9.x86_64 1/31 2023-11-25T04:53:56.670 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-2:18.0.0-7506.gec928368.el9.x86_64 2/31 2023-11-25T04:53:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-2:18.0.0-7506.gec928368.el9.x86_64 3/31 2023-11-25T04:53:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-selinux-2:18.0.0-7506.gec928368.el9.x86_64 4/31 2023-11-25T04:53:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-2:18.0.0-7506.gec928368.el9.x86_64 5/31 2023-11-25T04:53:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradosstriper1-2:18.0.0-7506.gec928368.el9.x86_6 6/31 2023-11-25T04:53:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librgw2-2:18.0.0-7506.gec928368.el9.x86_64 7/31 2023-11-25T04:53:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7506.gec928368.el9. 8/31 2023-11-25T04:53:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-common-2:18.0.0-7506.gec928368.el9.x8 9/31 2023-11-25T04:53:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-2:18.0.0-7506.gec928368.el9.x86_64 10/31 2023-11-25T04:53:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-2:18.0.0-7506.gec928368.el9.x86_64 11/31 2023-11-25T04:53:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-2:18.0.0-7506.gec928368.el9.x86_64 12/31 2023-11-25T04:53:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-2:18.0.0-7506.gec928368.el9.x86_64 13/31 2023-11-25T04:53:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-11-25T04:53:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-11-25T04:53:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-11-25T04:53:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-11-25T04:53:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-11-25T04:53:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-11-25T04:53:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-11-25T04:53:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-11-25T04:53:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-11-25T04:53:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-11-25T04:53:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-11-25T04:53:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-11-25T04:53:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-11-25T04:53:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-11-25T04:53:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:18.0.0-7506.gec928368.el9.x86_64 28/31 2023-11-25T04:53:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-11-25T04:53:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:18.0.0-7506.gec928368.el9.x86_64 30/31 2023-11-25T04:53:56.758 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:27 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:53:56.909 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:53:56.910 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:53:56.910 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2023-11-25T04:53:56.911 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:53:56.911 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:53:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:18.0.0-7506.gec928368.el9 ceph 41 M 2023-11-25T04:53:56.911 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2023-11-25T04:53:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-25T04:53:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:18.0.0-7506.gec928368.el9 ceph 163 k 2023-11-25T04:53:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-25T04:53:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-25T04:53:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-25T04:53:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:53:56.911 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:53:56.911 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:53:56.912 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2023-11-25T04:53:56.912 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:53:56.912 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 42 M 2023-11-25T04:53:56.912 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 173 M 2023-11-25T04:53:56.912 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:53:57.061 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 186 kB 00:00 2023-11-25T04:53:57.111 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2023-11-25T04:53:57.154 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.8 MB/s | 300 kB 00:00 2023-11-25T04:53:57.188 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-11-25T04:53:57.246 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): libcephsqlite-18.0.0-7506.gec928368.el9. 490 kB/s | 163 kB 00:00 2023-11-25T04:53:57.488 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-11-25T04:53:57.488 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:53:57.488 INFO:teuthology.orchestra.run.smithi007.stdout:Upgraded: 2023-11-25T04:53:57.488 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:57.488 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:57.488 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:53:57.488 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-25T04:53:57.488 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:57.489 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:57.489 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:57.489 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:57.489 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-25T04:53:57.489 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-25T04:53:57.489 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-25T04:53:57.489 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:57.489 INFO:teuthology.orchestra.run.smithi007.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-25T04:53:57.489 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-25T04:53:57.489 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-25T04:53:57.490 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:57.490 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-25T04:53:57.490 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:57.490 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-25T04:53:57.490 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-25T04:53:57.490 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-25T04:53:57.490 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:57.490 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:57.490 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:57.490 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-25T04:53:57.490 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:57.490 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:57.491 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:53:57.491 INFO:teuthology.orchestra.run.smithi007.stdout: re2-1:20211101-3.el9.x86_64 2023-11-25T04:53:57.491 INFO:teuthology.orchestra.run.smithi007.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-25T04:53:57.491 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:53:57.491 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:53:57.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-test 2023-11-25T04:53:58.394 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:28 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:53:58.555 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:53:58.556 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:53:58.556 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-25T04:53:58.556 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:53:58.556 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:53:58.557 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test x86_64 2:18.0.0-7506.gec928368.el9 ceph 41 M 2023-11-25T04:53:58.557 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-25T04:53:58.557 INFO:teuthology.orchestra.run.smithi007.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-25T04:53:58.557 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite x86_64 2:18.0.0-7506.gec928368.el9 ceph 163 k 2023-11-25T04:53:58.557 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-25T04:53:58.557 INFO:teuthology.orchestra.run.smithi007.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-25T04:53:58.557 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-25T04:53:58.557 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:53:58.557 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:53:58.557 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:53:58.557 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2023-11-25T04:53:58.557 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:53:58.558 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 42 M 2023-11-25T04:53:58.558 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 173 M 2023-11-25T04:53:58.558 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:53:58.701 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 186 kB 00:00 2023-11-25T04:53:58.742 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.3 MB/s | 217 kB 00:00 2023-11-25T04:53:58.784 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-11-25T04:53:58.818 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-11-25T04:53:58.868 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): libcephsqlite-18.0.0-7506.gec928368.el9. 528 kB/s | 163 kB 00:00 2023-11-25T04:53:59.055 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): ceph-test-18.0.0-7506.gec928368.el9.x86_ 19 MB/s | 41 MB 00:02 2023-11-25T04:53:59.058 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:53:59.058 INFO:teuthology.orchestra.run.smithi090.stdout:Total 20 MB/s | 42 MB 00:02 2023-11-25T04:53:59.059 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:53:59.078 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:53:59.078 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:53:59.429 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:53:59.430 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:53:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:54:00.094 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-11-25T04:54:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-11-25T04:54:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-11-25T04:54:00.231 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-11-25T04:54:00.259 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:18.0.0-7506.gec928368.el9.x86_64 5/6 2023-11-25T04:54:00.718 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): ceph-test-18.0.0-7506.gec928368.el9.x86_ 19 MB/s | 41 MB 00:02 2023-11-25T04:54:00.721 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:00.721 INFO:teuthology.orchestra.run.smithi007.stdout:Total 19 MB/s | 42 MB 00:02 2023-11-25T04:54:00.722 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:54:00.744 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:54:00.744 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:54:01.115 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:54:01.116 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:54:01.681 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:54:01.773 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-11-25T04:54:01.821 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-11-25T04:54:01.852 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-11-25T04:54:01.894 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-11-25T04:54:01.921 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephsqlite-2:18.0.0-7506.gec928368.el9.x86_64 5/6 2023-11-25T04:54:04.152 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7506.gec928368.el9.x86_64 5/6 2023-11-25T04:54:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:18.0.0-7506.gec928368.el9.x86_64 6/6 2023-11-25T04:54:06.007 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7506.gec928368.el9.x86_64 5/6 2023-11-25T04:54:06.082 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-2:18.0.0-7506.gec928368.el9.x86_64 6/6 2023-11-25T04:54:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-test-2:18.0.0-7506.gec928368.el9.x86_64 6/6 2023-11-25T04:54:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:18.0.0-7506.gec928368.el9.x86_64 1/6 2023-11-25T04:54:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:18.0.0-7506.gec928368.el9.x86_64 2/6 2023-11-25T04:54:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-11-25T04:54:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-11-25T04:54:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-11-25T04:54:07.284 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-11-25T04:54:07.285 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:07.285 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:54:07.285 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:07.285 INFO:teuthology.orchestra.run.smithi090.stdout: jq-1.6-16.el9.x86_64 2023-11-25T04:54:07.285 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:07.285 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-25T04:54:07.285 INFO:teuthology.orchestra.run.smithi090.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-25T04:54:07.285 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-25T04:54:07.285 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:07.285 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:54:07.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph 2023-11-25T04:54:08.186 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:39 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:54:08.252 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-test-2:18.0.0-7506.gec928368.el9.x86_64 6/6 2023-11-25T04:54:08.252 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-2:18.0.0-7506.gec928368.el9.x86_64 1/6 2023-11-25T04:54:08.252 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephsqlite-2:18.0.0-7506.gec928368.el9.x86_64 2/6 2023-11-25T04:54:08.252 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-11-25T04:54:08.253 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-11-25T04:54:08.253 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-11-25T04:54:08.347 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:54:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:========================================================================================= 2023-11-25T04:54:08.350 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2023-11-25T04:54:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:========================================================================================= 2023-11-25T04:54:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:54:08.350 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:18.0.0-7506.gec928368.el9 ceph 6.6 k 2023-11-25T04:54:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2023-11-25T04:54:08.350 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:18.0.0-7506.gec928368.el9 ceph 2.1 M 2023-11-25T04:54:08.350 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:18.0.0-7506.gec928368.el9 ceph 1.5 M 2023-11-25T04:54:08.351 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 243 k 2023-11-25T04:54:08.351 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:18.0.0-7506.gec928368.el9 ceph 4.4 M 2023-11-25T04:54:08.351 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:18.0.0-7506.gec928368.el9 ceph 18 M 2023-11-25T04:54:08.351 INFO:teuthology.orchestra.run.smithi090.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-11-25T04:54:08.351 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-11-25T04:54:08.351 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-11-25T04:54:08.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-25T04:54:08.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-25T04:54:08.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-25T04:54:08.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-25T04:54:08.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-11-25T04:54:08.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-25T04:54:08.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-25T04:54:08.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-25T04:54:08.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-25T04:54:08.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-25T04:54:08.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-25T04:54:08.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-25T04:54:08.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-25T04:54:08.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-25T04:54:08.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-25T04:54:08.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-25T04:54:08.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-25T04:54:08.353 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-25T04:54:08.353 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-25T04:54:08.353 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-11-25T04:54:08.353 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-25T04:54:08.353 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-25T04:54:08.353 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-25T04:54:08.353 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-25T04:54:08.353 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-25T04:54:08.353 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-25T04:54:08.353 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-25T04:54:08.353 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:08.353 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:54:08.353 INFO:teuthology.orchestra.run.smithi090.stdout:========================================================================================= 2023-11-25T04:54:08.354 INFO:teuthology.orchestra.run.smithi090.stdout:Install 36 Packages 2023-11-25T04:54:08.354 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:08.354 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 30 M 2023-11-25T04:54:08.354 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 111 M 2023-11-25T04:54:08.354 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:54:08.664 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-11-25T04:54:08.664 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:08.664 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:54:08.664 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:08.664 INFO:teuthology.orchestra.run.smithi007.stdout: jq-1.6-16.el9.x86_64 2023-11-25T04:54:08.664 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:08.664 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-25T04:54:08.664 INFO:teuthology.orchestra.run.smithi007.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-25T04:54:08.665 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-25T04:54:08.665 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:08.665 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:54:08.899 INFO:teuthology.orchestra.run.smithi090.stdout:(1/36): ceph-18.0.0-7506.gec928368.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2023-11-25T04:54:08.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph 2023-11-25T04:54:09.424 INFO:teuthology.orchestra.run.smithi090.stdout:(2/36): ceph-mgr-18.0.0-7506.gec928368.el9.x86_ 1.9 MB/s | 1.5 MB 00:00 2023-11-25T04:54:09.508 INFO:teuthology.orchestra.run.smithi090.stdout:(3/36): ceph-mds-18.0.0-7506.gec928368.el9.x86_ 2.5 MB/s | 2.1 MB 00:00 2023-11-25T04:54:09.562 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:39 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:54:09.608 INFO:teuthology.orchestra.run.smithi090.stdout:(4/36): ceph-mgr-modules-core-18.0.0-7506.gec92 2.4 MB/s | 243 kB 00:00 2023-11-25T04:54:09.717 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:54:09.717 INFO:teuthology.orchestra.run.smithi090.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-11-25T04:54:09.719 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2023-11-25T04:54:09.719 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-25T04:54:09.719 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2023-11-25T04:54:09.719 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:54:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: ceph x86_64 2:18.0.0-7506.gec928368.el9 ceph 6.6 k 2023-11-25T04:54:09.720 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-25T04:54:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds x86_64 2:18.0.0-7506.gec928368.el9 ceph 2.1 M 2023-11-25T04:54:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr x86_64 2:18.0.0-7506.gec928368.el9 ceph 1.5 M 2023-11-25T04:54:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 243 k 2023-11-25T04:54:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon x86_64 2:18.0.0-7506.gec928368.el9 ceph 4.4 M 2023-11-25T04:54:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd x86_64 2:18.0.0-7506.gec928368.el9 ceph 18 M 2023-11-25T04:54:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-11-25T04:54:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-11-25T04:54:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-11-25T04:54:09.720 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-25T04:54:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-25T04:54:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-25T04:54:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-25T04:54:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-11-25T04:54:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-25T04:54:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-25T04:54:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-25T04:54:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-25T04:54:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-25T04:54:09.721 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-25T04:54:09.722 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-25T04:54:09.722 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-25T04:54:09.722 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-25T04:54:09.722 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-25T04:54:09.722 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-25T04:54:09.722 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-25T04:54:09.722 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-25T04:54:09.722 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-25T04:54:09.723 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-11-25T04:54:09.723 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-25T04:54:09.723 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-25T04:54:09.723 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-25T04:54:09.723 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-25T04:54:09.723 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-25T04:54:09.723 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-25T04:54:09.723 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-25T04:54:09.723 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:09.723 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:54:09.723 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2023-11-25T04:54:09.724 INFO:teuthology.orchestra.run.smithi007.stdout:Install 36 Packages 2023-11-25T04:54:09.724 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:09.724 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 30 M 2023-11-25T04:54:09.724 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 111 M 2023-11-25T04:54:09.725 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:54:09.759 INFO:teuthology.orchestra.run.smithi090.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2023-11-25T04:54:09.792 INFO:teuthology.orchestra.run.smithi090.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2023-11-25T04:54:09.826 INFO:teuthology.orchestra.run.smithi090.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-11-25T04:54:09.859 INFO:teuthology.orchestra.run.smithi090.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-11-25T04:54:09.935 INFO:teuthology.orchestra.run.smithi090.stdout:(10/36): ceph-mon-18.0.0-7506.gec928368.el9.x86 4.2 MB/s | 4.4 MB 00:01 2023-11-25T04:54:09.960 INFO:teuthology.orchestra.run.smithi090.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 241 kB 00:00 2023-11-25T04:54:10.027 INFO:teuthology.orchestra.run.smithi090.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2023-11-25T04:54:10.053 INFO:teuthology.orchestra.run.smithi090.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2023-11-25T04:54:10.078 INFO:teuthology.orchestra.run.smithi090.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-11-25T04:54:10.103 INFO:teuthology.orchestra.run.smithi090.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-11-25T04:54:10.128 INFO:teuthology.orchestra.run.smithi090.stdout:(16/36): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-11-25T04:54:10.396 INFO:teuthology.orchestra.run.smithi090.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 76 kB/s | 22 kB 00:00 2023-11-25T04:54:10.404 INFO:teuthology.orchestra.run.smithi007.stdout:(1/36): ceph-18.0.0-7506.gec928368.el9.x86_64.r 30 kB/s | 6.6 kB 00:00 2023-11-25T04:54:10.529 INFO:teuthology.orchestra.run.smithi090.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 377 kB/s | 151 kB 00:00 2023-11-25T04:54:10.696 INFO:teuthology.orchestra.run.smithi090.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2023-11-25T04:54:10.713 INFO:teuthology.orchestra.run.smithi090.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 137 kB/s | 43 kB 00:00 2023-11-25T04:54:10.831 INFO:teuthology.orchestra.run.smithi090.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2023-11-25T04:54:10.881 INFO:teuthology.orchestra.run.smithi090.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.9 MB/s | 358 kB 00:00 2023-11-25T04:54:10.922 INFO:teuthology.orchestra.run.smithi090.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2023-11-25T04:54:10.930 INFO:teuthology.orchestra.run.smithi007.stdout:(2/36): ceph-mgr-18.0.0-7506.gec928368.el9.x86_ 2.0 MB/s | 1.5 MB 00:00 2023-11-25T04:54:11.080 INFO:teuthology.orchestra.run.smithi007.stdout:(3/36): ceph-mds-18.0.0-7506.gec928368.el9.x86_ 2.3 MB/s | 2.1 MB 00:00 2023-11-25T04:54:11.139 INFO:teuthology.orchestra.run.smithi090.stdout:(24/36): ceph-osd-18.0.0-7506.gec928368.el9.x86 10 MB/s | 18 MB 00:01 2023-11-25T04:54:11.156 INFO:teuthology.orchestra.run.smithi090.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 84 kB/s | 23 kB 00:00 2023-11-25T04:54:11.173 INFO:teuthology.orchestra.run.smithi090.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 77 kB/s | 19 kB 00:00 2023-11-25T04:54:11.180 INFO:teuthology.orchestra.run.smithi007.stdout:(4/36): ceph-mgr-modules-core-18.0.0-7506.gec92 2.4 MB/s | 243 kB 00:00 2023-11-25T04:54:11.274 INFO:teuthology.orchestra.run.smithi090.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 462 kB/s | 46 kB 00:00 2023-11-25T04:54:11.280 INFO:teuthology.orchestra.run.smithi007.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2023-11-25T04:54:11.323 INFO:teuthology.orchestra.run.smithi007.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2023-11-25T04:54:11.356 INFO:teuthology.orchestra.run.smithi007.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 949 kB/s | 32 kB 00:00 2023-11-25T04:54:11.374 INFO:teuthology.orchestra.run.smithi090.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 393 kB/s | 79 kB 00:00 2023-11-25T04:54:11.399 INFO:teuthology.orchestra.run.smithi090.stdout:(29/36): python3-jaraco-text-3.2.0-6.el9.noarch 76 kB/s | 20 kB 00:00 2023-11-25T04:54:11.423 INFO:teuthology.orchestra.run.smithi007.stdout:(8/36): ceph-mon-18.0.0-7506.gec928368.el9.x86_ 4.3 MB/s | 4.4 MB 00:01 2023-11-25T04:54:11.424 INFO:teuthology.orchestra.run.smithi090.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2023-11-25T04:54:11.448 INFO:teuthology.orchestra.run.smithi007.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 513 kB/s | 47 kB 00:00 2023-11-25T04:54:11.466 INFO:teuthology.orchestra.run.smithi090.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 177 kB/s | 16 kB 00:00 2023-11-25T04:54:11.473 INFO:teuthology.orchestra.run.smithi007.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 772 kB/s | 39 kB 00:00 2023-11-25T04:54:11.499 INFO:teuthology.orchestra.run.smithi007.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-11-25T04:54:11.517 INFO:teuthology.orchestra.run.smithi090.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 388 kB/s | 36 kB 00:00 2023-11-25T04:54:11.557 INFO:teuthology.orchestra.run.smithi007.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2023-11-25T04:54:11.575 INFO:teuthology.orchestra.run.smithi090.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 511 kB/s | 90 kB 00:00 2023-11-25T04:54:11.583 INFO:teuthology.orchestra.run.smithi007.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2023-11-25T04:54:11.608 INFO:teuthology.orchestra.run.smithi007.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-11-25T04:54:11.633 INFO:teuthology.orchestra.run.smithi007.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-11-25T04:54:11.651 INFO:teuthology.orchestra.run.smithi090.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2023-11-25T04:54:11.659 INFO:teuthology.orchestra.run.smithi007.stdout:(16/36): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-11-25T04:54:11.677 INFO:teuthology.orchestra.run.smithi090.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2023-11-25T04:54:11.701 INFO:teuthology.orchestra.run.smithi090.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 980 kB/s | 230 kB 00:00 2023-11-25T04:54:11.705 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:11.705 INFO:teuthology.orchestra.run.smithi090.stdout:Total 9.0 MB/s | 30 MB 00:03 2023-11-25T04:54:11.718 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:54:11.760 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:54:11.761 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:54:11.809 INFO:teuthology.orchestra.run.smithi007.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2023-11-25T04:54:11.926 INFO:teuthology.orchestra.run.smithi007.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 373 kB/s | 43 kB 00:00 2023-11-25T04:54:11.951 INFO:teuthology.orchestra.run.smithi007.stdout:(19/36): lua-devel-5.4.4-4.el9.x86_64.rpm 70 kB/s | 22 kB 00:00 2023-11-25T04:54:12.102 INFO:teuthology.orchestra.run.smithi007.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 980 kB/s | 172 kB 00:00 2023-11-25T04:54:12.187 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:54:12.189 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:54:12.194 INFO:teuthology.orchestra.run.smithi007.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2023-11-25T04:54:12.219 INFO:teuthology.orchestra.run.smithi007.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2023-11-25T04:54:12.344 INFO:teuthology.orchestra.run.smithi007.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 118 kB/s | 18 kB 00:00 2023-11-25T04:54:12.369 INFO:teuthology.orchestra.run.smithi007.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 184 kB/s | 23 kB 00:00 2023-11-25T04:54:12.645 INFO:teuthology.orchestra.run.smithi007.stdout:(25/36): ceph-osd-18.0.0-7506.gec928368.el9.x86 10 MB/s | 18 MB 00:01 2023-11-25T04:54:12.670 INFO:teuthology.orchestra.run.smithi007.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 60 kB/s | 19 kB 00:00 2023-11-25T04:54:12.720 INFO:teuthology.orchestra.run.smithi007.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 60 kB/s | 20 kB 00:00 2023-11-25T04:54:12.829 INFO:teuthology.orchestra.run.smithi007.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 253 kB/s | 46 kB 00:00 2023-11-25T04:54:12.888 INFO:teuthology.orchestra.run.smithi007.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 410 kB/s | 79 kB 00:00 2023-11-25T04:54:12.921 INFO:teuthology.orchestra.run.smithi007.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2023-11-25T04:54:12.972 INFO:teuthology.orchestra.run.smithi007.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 115 kB/s | 16 kB 00:00 2023-11-25T04:54:13.089 INFO:teuthology.orchestra.run.smithi007.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 254 kB/s | 36 kB 00:00 2023-11-25T04:54:13.114 INFO:teuthology.orchestra.run.smithi007.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 400 kB/s | 90 kB 00:00 2023-11-25T04:54:13.206 INFO:teuthology.orchestra.run.smithi007.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2023-11-25T04:54:13.273 INFO:teuthology.orchestra.run.smithi007.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 150 kB/s | 20 kB 00:00 2023-11-25T04:54:13.323 INFO:teuthology.orchestra.run.smithi007.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.8 MB/s | 427 kB 00:00 2023-11-25T04:54:13.326 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:13.326 INFO:teuthology.orchestra.run.smithi007.stdout:Total 8.4 MB/s | 30 MB 00:03 2023-11-25T04:54:13.338 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:54:13.380 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:54:13.380 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:54:13.550 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:54:13.622 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-11-25T04:54:13.661 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-11-25T04:54:13.692 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-11-25T04:54:13.763 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-11-25T04:54:13.801 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:54:13.801 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:54:13.807 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-11-25T04:54:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-11-25T04:54:13.933 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-11-25T04:54:14.011 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-11-25T04:54:14.060 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-11-25T04:54:14.141 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-11-25T04:54:14.195 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-11-25T04:54:14.234 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-11-25T04:54:14.291 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-11-25T04:54:14.347 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-11-25T04:54:14.544 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-11-25T04:54:14.589 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-11-25T04:54:14.628 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-11-25T04:54:14.662 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-11-25T04:54:14.694 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-11-25T04:54:14.762 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-11-25T04:54:14.804 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-11-25T04:54:14.838 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-11-25T04:54:14.879 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-11-25T04:54:14.922 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-11-25T04:54:14.957 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-11-25T04:54:15.044 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-11-25T04:54:15.080 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-11-25T04:54:15.160 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-11-25T04:54:15.172 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:54:15.238 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-11-25T04:54:15.280 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-11-25T04:54:15.307 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-11-25T04:54:15.308 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-11-25T04:54:15.380 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-11-25T04:54:15.415 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-11-25T04:54:15.433 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-11-25T04:54:15.448 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-11-25T04:54:15.543 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-11-25T04:54:15.587 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7506.gec928368.el9. 31/36 2023-11-25T04:54:15.612 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:18.0.0-7506.gec928368.el9.x86_64 32/36 2023-11-25T04:54:15.620 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-11-25T04:54:15.636 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7506.gec928368.el9.x86_64 32/36 2023-11-25T04:54:15.636 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:54:15.636 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-25T04:54:15.636 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-25T04:54:15.636 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-25T04:54:15.636 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:15.676 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-11-25T04:54:15.749 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-11-25T04:54:15.802 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-11-25T04:54:15.840 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-11-25T04:54:15.898 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-11-25T04:54:15.953 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-11-25T04:54:16.159 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-11-25T04:54:16.219 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-11-25T04:54:16.258 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-11-25T04:54:16.292 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-11-25T04:54:16.322 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-11-25T04:54:16.401 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-11-25T04:54:16.432 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-11-25T04:54:16.466 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-11-25T04:54:16.506 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-11-25T04:54:16.550 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-11-25T04:54:16.584 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-11-25T04:54:16.676 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-11-25T04:54:16.715 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-11-25T04:54:16.790 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-11-25T04:54:16.931 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-11-25T04:54:17.062 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-11-25T04:54:17.225 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7506.gec928368.el9. 31/36 2023-11-25T04:54:17.241 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:18.0.0-7506.gec928368.el9.x86_64 33/36 2023-11-25T04:54:17.253 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-2:18.0.0-7506.gec928368.el9.x86_64 32/36 2023-11-25T04:54:17.268 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:18.0.0-7506.gec928368.el9.x86_64 33/36 2023-11-25T04:54:17.268 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:54:17.268 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-25T04:54:17.268 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-25T04:54:17.268 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-25T04:54:17.268 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:17.279 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7506.gec928368.el9.x86_64 32/36 2023-11-25T04:54:17.279 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:54:17.279 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-25T04:54:17.279 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-25T04:54:17.279 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-25T04:54:17.279 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:17.725 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:18.0.0-7506.gec928368.el9.x86_64 34/36 2023-11-25T04:54:17.748 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:18.0.0-7506.gec928368.el9.x86_64 34/36 2023-11-25T04:54:17.748 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:54:17.748 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-25T04:54:17.748 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-25T04:54:17.748 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-25T04:54:17.748 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:17.942 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:18.0.0-7506.gec928368.el9.x86_64 35/36 2023-11-25T04:54:17.965 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:18.0.0-7506.gec928368.el9.x86_64 35/36 2023-11-25T04:54:17.965 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:54:17.965 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-25T04:54:17.965 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-25T04:54:17.965 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-25T04:54:17.965 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:18.023 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:18.0.0-7506.gec928368.el9.x86_64 36/36 2023-11-25T04:54:18.904 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-2:18.0.0-7506.gec928368.el9.x86_64 33/36 2023-11-25T04:54:18.934 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-2:18.0.0-7506.gec928368.el9.x86_64 33/36 2023-11-25T04:54:18.934 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:54:18.934 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-25T04:54:18.934 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-25T04:54:18.934 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-25T04:54:18.935 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:19.423 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-2:18.0.0-7506.gec928368.el9.x86_64 34/36 2023-11-25T04:54:19.449 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-2:18.0.0-7506.gec928368.el9.x86_64 34/36 2023-11-25T04:54:19.449 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:54:19.449 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-25T04:54:19.449 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-25T04:54:19.449 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-25T04:54:19.449 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:19.673 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-2:18.0.0-7506.gec928368.el9.x86_64 35/36 2023-11-25T04:54:19.698 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-2:18.0.0-7506.gec928368.el9.x86_64 35/36 2023-11-25T04:54:19.698 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:54:19.698 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-25T04:54:19.698 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-25T04:54:19.698 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-25T04:54:19.699 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:19.756 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-2:18.0.0-7506.gec928368.el9.x86_64 36/36 2023-11-25T04:54:20.229 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-2:18.0.0-7506.gec928368.el9.x86_64 36/36 2023-11-25T04:54:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:18.0.0-7506.gec928368.el9.x86_64 1/36 2023-11-25T04:54:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:18.0.0-7506.gec928368.el9.x86_64 2/36 2023-11-25T04:54:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:18.0.0-7506.gec928368.el9.x86_64 3/36 2023-11-25T04:54:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:18.0.0-7506.gec928368.el9.x86_64 4/36 2023-11-25T04:54:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:18.0.0-7506.gec928368.el9.x86_64 5/36 2023-11-25T04:54:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7506.gec928368.el9. 6/36 2023-11-25T04:54:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-11-25T04:54:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-11-25T04:54:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-11-25T04:54:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-11-25T04:54:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-11-25T04:54:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-11-25T04:54:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-11-25T04:54:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-11-25T04:54:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-11-25T04:54:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-11-25T04:54:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-11-25T04:54:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-11-25T04:54:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-11-25T04:54:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-11-25T04:54:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-11-25T04:54:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-11-25T04:54:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-11-25T04:54:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-11-25T04:54:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-11-25T04:54:20.233 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-11-25T04:54:20.233 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-11-25T04:54:20.234 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-11-25T04:54:20.234 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-11-25T04:54:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-11-25T04:54:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-11-25T04:54:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-11-25T04:54:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-11-25T04:54:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-11-25T04:54:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-11-25T04:54:21.296 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-11-25T04:54:21.296 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:21.296 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:54:21.296 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:21.296 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:21.297 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:21.297 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:21.297 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:21.297 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:21.297 INFO:teuthology.orchestra.run.smithi090.stdout: lua-5.4.4-4.el9.x86_64 2023-11-25T04:54:21.297 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-11-25T04:54:21.297 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks-3.9.2-1.el9.noarch 2023-11-25T04:54:21.297 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-25T04:54:21.297 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-25T04:54:21.297 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-25T04:54:21.297 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-25T04:54:21.297 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-25T04:54:21.297 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-25T04:54:21.298 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-25T04:54:21.298 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-25T04:54:21.298 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-25T04:54:21.298 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-25T04:54:21.298 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-25T04:54:21.298 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-25T04:54:21.298 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-25T04:54:21.298 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-25T04:54:21.298 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-25T04:54:21.298 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply-3.11-14.el9.noarch 2023-11-25T04:54:21.298 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-25T04:54:21.298 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-25T04:54:21.298 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-25T04:54:21.299 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-25T04:54:21.299 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-25T04:54:21.299 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-25T04:54:21.299 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-25T04:54:21.299 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-25T04:54:21.299 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-25T04:54:21.299 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-25T04:54:21.299 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-25T04:54:21.299 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:21.299 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:54:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-2:18.0.0-7506.gec928368.el9.x86_64 36/36 2023-11-25T04:54:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-2:18.0.0-7506.gec928368.el9.x86_64 1/36 2023-11-25T04:54:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-2:18.0.0-7506.gec928368.el9.x86_64 2/36 2023-11-25T04:54:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-2:18.0.0-7506.gec928368.el9.x86_64 3/36 2023-11-25T04:54:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-2:18.0.0-7506.gec928368.el9.x86_64 4/36 2023-11-25T04:54:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-2:18.0.0-7506.gec928368.el9.x86_64 5/36 2023-11-25T04:54:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7506.gec928368.el9. 6/36 2023-11-25T04:54:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-11-25T04:54:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-11-25T04:54:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-11-25T04:54:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-11-25T04:54:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-11-25T04:54:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-11-25T04:54:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-11-25T04:54:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-11-25T04:54:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-11-25T04:54:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-11-25T04:54:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-11-25T04:54:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-11-25T04:54:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-11-25T04:54:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-11-25T04:54:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-11-25T04:54:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-11-25T04:54:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-11-25T04:54:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-11-25T04:54:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-11-25T04:54:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-11-25T04:54:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-11-25T04:54:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-11-25T04:54:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-11-25T04:54:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-11-25T04:54:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-11-25T04:54:21.654 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-11-25T04:54:21.654 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-11-25T04:54:21.654 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-11-25T04:54:21.654 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-11-25T04:54:21.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-base 2023-11-25T04:54:22.312 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:53 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:54:22.407 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-base-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:54:22.457 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:54:22.458 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2023-11-25T04:54:22.458 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:54:22.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install cephadm 2023-11-25T04:54:22.618 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-11-25T04:54:22.618 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:22.618 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:54:22.618 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:22.619 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:22.619 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:22.619 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:22.619 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:22.619 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:22.619 INFO:teuthology.orchestra.run.smithi007.stdout: lua-5.4.4-4.el9.x86_64 2023-11-25T04:54:22.620 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-11-25T04:54:22.620 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks-3.9.2-1.el9.noarch 2023-11-25T04:54:22.620 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-25T04:54:22.620 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-25T04:54:22.620 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-25T04:54:22.620 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-25T04:54:22.620 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-25T04:54:22.620 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-25T04:54:22.621 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-25T04:54:22.621 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-25T04:54:22.621 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-25T04:54:22.621 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-25T04:54:22.621 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-25T04:54:22.621 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-25T04:54:22.621 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-25T04:54:22.621 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-25T04:54:22.622 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-25T04:54:22.622 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply-3.11-14.el9.noarch 2023-11-25T04:54:22.622 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-25T04:54:22.622 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-25T04:54:22.622 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-25T04:54:22.622 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-25T04:54:22.623 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-25T04:54:22.623 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-25T04:54:22.623 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-25T04:54:22.623 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-25T04:54:22.623 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-25T04:54:22.623 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-25T04:54:22.624 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-25T04:54:22.624 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:22.624 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:54:22.861 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base 2023-11-25T04:54:23.093 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:54 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:54:23.236 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:54:23.237 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:54:23.237 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2023-11-25T04:54:23.237 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:54:23.237 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:54:23.237 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 776 k 2023-11-25T04:54:23.237 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:23.237 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:54:23.237 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:54:23.237 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T04:54:23.237 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:23.237 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 776 k 2023-11-25T04:54:23.237 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 789 k 2023-11-25T04:54:23.238 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:54:23.463 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:53 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:54:23.563 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-base-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:54:23.615 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:54:23.616 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-25T04:54:23.616 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:54:23.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install cephadm 2023-11-25T04:54:23.680 INFO:teuthology.orchestra.run.smithi090.stdout:cephadm-18.0.0-7506.gec928368.el9.noarch.rpm 1.7 MB/s | 776 kB 00:00 2023-11-25T04:54:23.681 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:23.681 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.7 MB/s | 776 kB 00:00 2023-11-25T04:54:23.681 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:54:23.686 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:54:23.686 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:54:23.692 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:54:23.693 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:54:23.767 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:54:24.237 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:18.0.0-7506.gec928368.el9.noarch 1/1 2023-11-25T04:54:24.279 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:54 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:54:24.329 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:18.0.0-7506.gec928368.el9.noarch 1/1 2023-11-25T04:54:24.427 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:54:24.428 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:54:24.428 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-25T04:54:24.428 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:54:24.429 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:54:24.429 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 776 k 2023-11-25T04:54:24.429 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:24.429 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:54:24.429 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:54:24.429 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T04:54:24.429 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:24.429 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 776 k 2023-11-25T04:54:24.429 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 789 k 2023-11-25T04:54:24.429 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:54:24.698 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm-18.0.0-7506.gec928368.el9.noarch.rpm 2.8 MB/s | 776 kB 00:00 2023-11-25T04:54:24.698 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:24.698 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.8 MB/s | 776 kB 00:00 2023-11-25T04:54:24.698 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:54:24.704 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:54:24.704 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:54:24.710 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:54:24.710 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:54:24.788 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:54:24.805 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:18.0.0-7506.gec928368.el9.noarch 1/1 2023-11-25T04:54:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:18.0.0-7506.gec928368.el9.noarch 1/1 2023-11-25T04:54:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:25.096 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:54:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:25.097 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:25.097 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:54:25.303 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:18.0.0-7506.gec928368.el9.noarch 1/1 2023-11-25T04:54:25.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-immutable-object-cache 2023-11-25T04:54:25.396 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephadm-2:18.0.0-7506.gec928368.el9.noarch 1/1 2023-11-25T04:54:25.761 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:18.0.0-7506.gec928368.el9.noarch 1/1 2023-11-25T04:54:25.905 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:56 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:54:26.050 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:54:26.050 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:54:26.050 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T04:54:26.051 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:54:26.051 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:54:26.051 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7506.gec928368.el9 ceph 141 k 2023-11-25T04:54:26.051 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:26.051 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:54:26.051 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:54:26.051 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T04:54:26.051 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:26.052 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 141 k 2023-11-25T04:54:26.052 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 395 k 2023-11-25T04:54:26.052 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:54:26.063 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephadm-2:18.0.0-7506.gec928368.el9.noarch 1/1 2023-11-25T04:54:26.063 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:26.063 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:54:26.063 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:26.063 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:26.063 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:54:26.250 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-immutable-object-cache-18.0.0-7506.gec9283 711 kB/s | 141 kB 00:00 2023-11-25T04:54:26.251 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:26.251 INFO:teuthology.orchestra.run.smithi090.stdout:Total 705 kB/s | 141 kB 00:00 2023-11-25T04:54:26.251 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:54:26.258 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:54:26.259 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:54:26.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-immutable-object-cache 2023-11-25T04:54:26.310 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:54:26.310 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:54:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:54:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7506.gec928368. 1/1 2023-11-25T04:54:26.535 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7506.gec928368. 1/1 2023-11-25T04:54:26.535 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:54:26.535 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-25T04:54:26.536 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:26.881 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:56 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:54:27.032 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:54:27.033 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:54:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T04:54:27.033 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:54:27.033 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:54:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7506.gec928368.el9 ceph 141 k 2023-11-25T04:54:27.033 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:27.033 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:54:27.034 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:54:27.034 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T04:54:27.034 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:27.034 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 141 k 2023-11-25T04:54:27.034 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 395 k 2023-11-25T04:54:27.034 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:54:27.395 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-immutable-object-cache-18.0.0-7506.gec9283 390 kB/s | 141 kB 00:00 2023-11-25T04:54:27.395 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:27.396 INFO:teuthology.orchestra.run.smithi007.stdout:Total 389 kB/s | 141 kB 00:00 2023-11-25T04:54:27.396 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:54:27.404 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:54:27.404 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:54:27.458 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:54:27.459 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:54:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7506.gec928368. 1/1 2023-11-25T04:54:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:27.565 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:54:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:27.565 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:54:27.604 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:54:27.671 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7506.gec928368. 1/1 2023-11-25T04:54:27.697 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7506.gec928368. 1/1 2023-11-25T04:54:27.697 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:54:27.697 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-25T04:54:27.697 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:27.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr 2023-11-25T04:54:28.434 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:59 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:54:28.532 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-mgr-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:54:28.582 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:54:28.583 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2023-11-25T04:54:28.583 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:54:28.649 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-dashboard 2023-11-25T04:54:28.653 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7506.gec928368. 1/1 2023-11-25T04:54:28.653 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:28.653 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:54:28.653 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:54:28.654 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:28.654 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:54:28.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr 2023-11-25T04:54:29.222 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:00 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:54:29.366 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:54:29.367 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:54:29.367 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2023-11-25T04:54:29.368 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:54:29.368 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:54:29.368 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 1.8 M 2023-11-25T04:54:29.368 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2023-11-25T04:54:29.368 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 28 k 2023-11-25T04:54:29.368 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 15 k 2023-11-25T04:54:29.368 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-25T04:54:29.368 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-25T04:54:29.368 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-25T04:54:29.368 INFO:teuthology.orchestra.run.smithi090.stdout:Installing weak dependencies: 2023-11-25T04:54:29.368 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-25T04:54:29.368 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:29.369 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:54:29.369 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:54:29.369 INFO:teuthology.orchestra.run.smithi090.stdout:Install 7 Packages 2023-11-25T04:54:29.369 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:29.369 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.1 M 2023-11-25T04:54:29.369 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 8.7 M 2023-11-25T04:54:29.369 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:54:29.478 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:59 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:54:29.577 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-mgr-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:54:29.629 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:54:29.630 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-25T04:54:29.630 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:54:29.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-dashboard 2023-11-25T04:54:29.742 INFO:teuthology.orchestra.run.smithi090.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7506.gec9 98 kB/s | 28 kB 00:00 2023-11-25T04:54:29.759 INFO:teuthology.orchestra.run.smithi090.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7506.gec92 50 kB/s | 15 kB 00:00 2023-11-25T04:54:29.960 INFO:teuthology.orchestra.run.smithi090.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 42 kB/s | 9.0 kB 00:00 2023-11-25T04:54:30.069 INFO:teuthology.orchestra.run.smithi090.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 132 kB/s | 41 kB 00:00 2023-11-25T04:54:30.094 INFO:teuthology.orchestra.run.smithi090.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 230 kB/s | 31 kB 00:00 2023-11-25T04:54:30.269 INFO:teuthology.orchestra.run.smithi090.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 943 kB/s | 188 kB 00:00 2023-11-25T04:54:30.295 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:00 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:54:30.428 INFO:teuthology.orchestra.run.smithi090.stdout:(7/7): ceph-mgr-dashboard-18.0.0-7506.gec928368 1.8 MB/s | 1.8 MB 00:00 2023-11-25T04:54:30.431 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:30.431 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.9 MB/s | 2.1 MB 00:01 2023-11-25T04:54:30.431 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:54:30.439 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:54:30.439 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:54:30.446 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:54:30.447 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:54:30.447 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-25T04:54:30.448 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:54:30.448 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:54:30.448 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 1.8 M 2023-11-25T04:54:30.448 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-25T04:54:30.448 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 28 k 2023-11-25T04:54:30.448 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 15 k 2023-11-25T04:54:30.448 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-25T04:54:30.448 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-25T04:54:30.448 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-25T04:54:30.448 INFO:teuthology.orchestra.run.smithi007.stdout:Installing weak dependencies: 2023-11-25T04:54:30.448 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-25T04:54:30.448 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:30.449 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:54:30.449 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:54:30.449 INFO:teuthology.orchestra.run.smithi007.stdout:Install 7 Packages 2023-11-25T04:54:30.449 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:30.450 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.1 M 2023-11-25T04:54:30.450 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 8.7 M 2023-11-25T04:54:30.450 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:54:30.503 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:54:30.504 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:54:30.682 INFO:teuthology.orchestra.run.smithi007.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7506.gec9 156 kB/s | 28 kB 00:00 2023-11-25T04:54:30.791 INFO:teuthology.orchestra.run.smithi007.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7506.gec92 52 kB/s | 15 kB 00:00 2023-11-25T04:54:30.801 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:54:30.825 INFO:teuthology.orchestra.run.smithi007.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 64 kB/s | 9.0 kB 00:00 2023-11-25T04:54:30.854 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-11-25T04:54:30.888 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-11-25T04:54:30.917 INFO:teuthology.orchestra.run.smithi007.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 457 kB/s | 31 kB 00:00 2023-11-25T04:54:30.935 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-11-25T04:54:30.967 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-11-25T04:54:30.998 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7506.gec928368.el9.n 5/7 2023-11-25T04:54:31.009 INFO:teuthology.orchestra.run.smithi007.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 188 kB/s | 41 kB 00:00 2023-11-25T04:54:31.034 INFO:teuthology.orchestra.run.smithi007.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2023-11-25T04:54:31.076 INFO:teuthology.orchestra.run.smithi007.stdout:(7/7): ceph-mgr-dashboard-18.0.0-7506.gec928368 3.1 MB/s | 1.8 MB 00:00 2023-11-25T04:54:31.079 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:31.079 INFO:teuthology.orchestra.run.smithi007.stdout:Total 3.3 MB/s | 2.1 MB 00:00 2023-11-25T04:54:31.079 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:54:31.087 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:54:31.088 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:54:31.160 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:54:31.161 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:54:31.232 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7506.gec928368.el9. 6/7 2023-11-25T04:54:31.267 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7506.gec928368.el9.noarc 7/7 2023-11-25T04:54:31.460 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:54:31.519 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-11-25T04:54:31.553 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-11-25T04:54:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-11-25T04:54:31.631 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-11-25T04:54:31.670 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7506.gec928368.el9.n 5/7 2023-11-25T04:54:31.917 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7506.gec928368.el9. 6/7 2023-11-25T04:54:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7506.gec928368.el9.noarc 7/7 2023-11-25T04:54:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7506.gec928368.el9. 1/7 2023-11-25T04:54:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7506.gec928368.el9.noarc 2/7 2023-11-25T04:54:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7506.gec928368.el9.n 3/7 2023-11-25T04:54:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-11-25T04:54:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-11-25T04:54:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-11-25T04:54:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7506.gec928368.el9.noarc 7/7 2023-11-25T04:54:32.248 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-11-25T04:54:32.248 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:32.249 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:54:32.249 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:32.249 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:32.249 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:32.249 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-25T04:54:32.249 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-25T04:54:32.249 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-25T04:54:32.249 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-25T04:54:32.249 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:32.249 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:54:32.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-25T04:54:32.524 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7506.gec928368.el9.noarc 7/7 2023-11-25T04:54:32.524 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7506.gec928368.el9. 1/7 2023-11-25T04:54:32.524 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7506.gec928368.el9.noarc 2/7 2023-11-25T04:54:32.525 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7506.gec928368.el9.n 3/7 2023-11-25T04:54:32.525 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-11-25T04:54:32.525 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-11-25T04:54:32.525 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-11-25T04:54:32.853 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-11-25T04:54:32.854 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:32.854 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:54:32.854 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:32.854 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:32.854 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:32.854 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-25T04:54:32.854 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-25T04:54:32.854 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-25T04:54:32.854 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-25T04:54:32.854 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:32.854 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:54:33.083 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-25T04:54:33.092 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:04 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:54:33.238 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:54:33.239 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2023-11-25T04:54:33.239 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2023-11-25T04:54:33.239 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2023-11-25T04:54:33.240 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:54:33.240 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 7.4 M 2023-11-25T04:54:33.240 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2023-11-25T04:54:33.240 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-25T04:54:33.240 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-25T04:54:33.240 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-25T04:54:33.240 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-25T04:54:33.240 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-25T04:54:33.240 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-25T04:54:33.240 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-25T04:54:33.241 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-11-25T04:54:33.241 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-25T04:54:33.241 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-25T04:54:33.241 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-25T04:54:33.241 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:33.241 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:54:33.241 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2023-11-25T04:54:33.241 INFO:teuthology.orchestra.run.smithi090.stdout:Install 12 Packages 2023-11-25T04:54:33.241 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:33.242 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 37 M 2023-11-25T04:54:33.242 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 209 M 2023-11-25T04:54:33.242 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:54:33.571 INFO:teuthology.orchestra.run.smithi090.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 184 kB/s | 32 kB 00:00 2023-11-25T04:54:33.614 INFO:teuthology.orchestra.run.smithi090.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 420 kB/s | 17 kB 00:00 2023-11-25T04:54:33.655 INFO:teuthology.orchestra.run.smithi090.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 829 kB/s | 34 kB 00:00 2023-11-25T04:54:33.687 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:03 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:54:33.798 INFO:teuthology.orchestra.run.smithi090.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.5 MB/s | 3.0 MB 00:00 2023-11-25T04:54:33.839 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:54:33.840 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2023-11-25T04:54:33.840 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-25T04:54:33.840 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2023-11-25T04:54:33.840 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:54:33.840 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 7.4 M 2023-11-25T04:54:33.840 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-25T04:54:33.840 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-25T04:54:33.841 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-25T04:54:33.841 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-25T04:54:33.841 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-25T04:54:33.841 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-25T04:54:33.841 INFO:teuthology.orchestra.run.smithi007.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-25T04:54:33.841 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-25T04:54:33.841 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-11-25T04:54:33.841 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-25T04:54:33.841 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-25T04:54:33.841 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-25T04:54:33.841 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:33.841 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:54:33.842 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2023-11-25T04:54:33.842 INFO:teuthology.orchestra.run.smithi007.stdout:Install 12 Packages 2023-11-25T04:54:33.842 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:33.842 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 37 M 2023-11-25T04:54:33.842 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 209 M 2023-11-25T04:54:33.842 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:54:34.056 INFO:teuthology.orchestra.run.smithi090.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2023-11-25T04:54:34.157 INFO:teuthology.orchestra.run.smithi090.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 4.8 MB 00:00 2023-11-25T04:54:34.185 INFO:teuthology.orchestra.run.smithi090.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2023-11-25T04:54:34.281 INFO:teuthology.orchestra.run.smithi007.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 202 kB/s | 32 kB 00:00 2023-11-25T04:54:34.282 INFO:teuthology.orchestra.run.smithi007.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 258 kB/s | 17 kB 00:00 2023-11-25T04:54:34.333 INFO:teuthology.orchestra.run.smithi007.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 512 kB/s | 34 kB 00:00 2023-11-25T04:54:34.400 INFO:teuthology.orchestra.run.smithi090.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2023-11-25T04:54:34.634 INFO:teuthology.orchestra.run.smithi090.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2023-11-25T04:54:34.642 INFO:teuthology.orchestra.run.smithi007.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2023-11-25T04:54:34.659 INFO:teuthology.orchestra.run.smithi090.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 761 kB/s | 195 kB 00:00 2023-11-25T04:54:34.726 INFO:teuthology.orchestra.run.smithi007.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2023-11-25T04:54:34.818 INFO:teuthology.orchestra.run.smithi007.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 4.8 MB 00:00 2023-11-25T04:54:34.843 INFO:teuthology.orchestra.run.smithi007.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.9 MB/s | 217 kB 00:00 2023-11-25T04:54:34.851 INFO:teuthology.orchestra.run.smithi090.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2023-11-25T04:54:35.201 INFO:teuthology.orchestra.run.smithi007.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2023-11-25T04:54:35.227 INFO:teuthology.orchestra.run.smithi007.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2023-11-25T04:54:35.252 INFO:teuthology.orchestra.run.smithi007.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.9 MB/s | 195 kB 00:00 2023-11-25T04:54:35.344 INFO:teuthology.orchestra.run.smithi007.stdout:(11/12): ceph-mgr-diskprediction-local-18.0.0-7 5.7 MB/s | 7.4 MB 00:01 2023-11-25T04:54:35.502 INFO:teuthology.orchestra.run.smithi007.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 892 kB/s | 245 kB 00:00 2023-11-25T04:54:35.506 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:35.506 INFO:teuthology.orchestra.run.smithi007.stdout:Total 22 MB/s | 37 MB 00:01 2023-11-25T04:54:35.520 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:54:35.546 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:54:35.546 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:54:35.810 INFO:teuthology.orchestra.run.smithi090.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 3.1 MB/s | 7.4 MB 00:02 2023-11-25T04:54:35.812 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:35.813 INFO:teuthology.orchestra.run.smithi090.stdout:Total 14 MB/s | 37 MB 00:02 2023-11-25T04:54:35.826 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:54:35.851 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:54:35.851 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:54:35.963 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:54:35.964 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:54:36.246 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:54:36.247 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:54:36.715 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:54:36.796 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-11-25T04:54:36.823 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-11-25T04:54:36.920 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-11-25T04:54:36.946 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:54:36.960 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-11-25T04:54:37.032 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-11-25T04:54:37.064 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-11-25T04:54:37.151 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-25T04:54:37.157 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-11-25T04:54:37.176 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-25T04:54:37.210 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-11-25T04:54:37.255 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-11-25T04:54:37.397 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-25T04:54:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-25T04:54:37.511 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-11-25T04:54:37.952 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-11-25T04:54:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-11-25T04:54:38.168 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-11-25T04:54:38.281 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-11-25T04:54:39.384 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-11-25T04:54:39.514 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-11-25T04:54:40.469 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-11-25T04:54:40.493 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7506.gec928 12/12 2023-11-25T04:54:40.567 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-11-25T04:54:40.625 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7506.gec928 12/12 2023-11-25T04:54:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7506.gec928 12/12 2023-11-25T04:54:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7506.gec928 1/12 2023-11-25T04:54:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-11-25T04:54:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-11-25T04:54:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-11-25T04:54:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-25T04:54:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-25T04:54:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-11-25T04:54:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-11-25T04:54:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-11-25T04:54:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-11-25T04:54:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-11-25T04:54:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-11-25T04:54:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:43.537 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:54:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-25T04:54:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-25T04:54:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-25T04:54:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-25T04:54:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-25T04:54:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-25T04:54:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-25T04:54:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-25T04:54:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-25T04:54:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-25T04:54:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-25T04:54:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:43.538 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:54:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7506.gec928 12/12 2023-11-25T04:54:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7506.gec928 1/12 2023-11-25T04:54:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-11-25T04:54:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-11-25T04:54:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-11-25T04:54:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-25T04:54:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-25T04:54:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-11-25T04:54:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-11-25T04:54:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-11-25T04:54:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-11-25T04:54:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-11-25T04:54:43.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-rook 2023-11-25T04:54:44.266 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-11-25T04:54:44.266 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:44.266 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:54:44.266 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-25T04:54:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-25T04:54:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-25T04:54:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-25T04:54:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-25T04:54:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-25T04:54:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-25T04:54:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-25T04:54:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-25T04:54:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-25T04:54:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-25T04:54:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:44.267 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:54:44.511 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:14 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:54:44.662 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:54:44.663 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2023-11-25T04:54:44.663 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-25T04:54:44.663 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2023-11-25T04:54:44.664 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:54:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 50 k 2023-11-25T04:54:44.664 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-25T04:54:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-25T04:54:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-25T04:54:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-11-25T04:54:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-25T04:54:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-25T04:54:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-25T04:54:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-25T04:54:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-25T04:54:44.665 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-25T04:54:44.665 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-25T04:54:44.665 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-25T04:54:44.665 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-25T04:54:44.665 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:44.665 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:54:44.665 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2023-11-25T04:54:44.665 INFO:teuthology.orchestra.run.smithi007.stdout:Install 13 Packages 2023-11-25T04:54:44.665 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:44.666 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.0 M 2023-11-25T04:54:44.666 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 26 M 2023-11-25T04:54:44.666 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:54:44.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-rook 2023-11-25T04:54:44.929 INFO:teuthology.orchestra.run.smithi007.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 115 kB/s | 18 kB 00:00 2023-11-25T04:54:44.954 INFO:teuthology.orchestra.run.smithi007.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 134 kB/s | 24 kB 00:00 2023-11-25T04:54:45.013 INFO:teuthology.orchestra.run.smithi007.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2023-11-25T04:54:45.038 INFO:teuthology.orchestra.run.smithi007.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2023-11-25T04:54:45.063 INFO:teuthology.orchestra.run.smithi007.stdout:(5/13): ceph-mgr-rook-18.0.0-7506.gec928368.el9 172 kB/s | 50 kB 00:00 2023-11-25T04:54:45.089 INFO:teuthology.orchestra.run.smithi007.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2023-11-25T04:54:45.114 INFO:teuthology.orchestra.run.smithi007.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 568 kB/s | 43 kB 00:00 2023-11-25T04:54:45.318 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:16 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:54:45.356 INFO:teuthology.orchestra.run.smithi007.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 62 kB/s | 15 kB 00:00 2023-11-25T04:54:45.456 INFO:teuthology.orchestra.run.smithi007.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 82 kB/s | 32 kB 00:00 2023-11-25T04:54:45.464 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:54:45.466 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2023-11-25T04:54:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2023-11-25T04:54:45.466 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2023-11-25T04:54:45.466 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:54:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 50 k 2023-11-25T04:54:45.467 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2023-11-25T04:54:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-25T04:54:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-25T04:54:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-11-25T04:54:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-25T04:54:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-25T04:54:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-25T04:54:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-25T04:54:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-25T04:54:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-25T04:54:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-25T04:54:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-25T04:54:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-25T04:54:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:45.469 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:54:45.469 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2023-11-25T04:54:45.469 INFO:teuthology.orchestra.run.smithi090.stdout:Install 13 Packages 2023-11-25T04:54:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:45.470 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.0 M 2023-11-25T04:54:45.471 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 26 M 2023-11-25T04:54:45.471 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:54:45.540 INFO:teuthology.orchestra.run.smithi007.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 464 kB/s | 197 kB 00:00 2023-11-25T04:54:45.582 INFO:teuthology.orchestra.run.smithi007.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 472 kB/s | 59 kB 00:00 2023-11-25T04:54:45.640 INFO:teuthology.orchestra.run.smithi007.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 895 kB/s | 90 kB 00:00 2023-11-25T04:54:45.716 INFO:teuthology.orchestra.run.smithi007.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2023-11-25T04:54:45.720 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:45.720 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2023-11-25T04:54:45.720 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:54:45.735 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:54:45.735 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:54:45.824 INFO:teuthology.orchestra.run.smithi090.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 131 kB/s | 24 kB 00:00 2023-11-25T04:54:45.845 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:54:45.845 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:54:45.849 INFO:teuthology.orchestra.run.smithi090.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 85 kB/s | 18 kB 00:00 2023-11-25T04:54:45.949 INFO:teuthology.orchestra.run.smithi090.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 174 kB 00:00 2023-11-25T04:54:45.983 INFO:teuthology.orchestra.run.smithi090.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 989 kB/s | 132 kB 00:00 2023-11-25T04:54:46.025 INFO:teuthology.orchestra.run.smithi090.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2023-11-25T04:54:46.050 INFO:teuthology.orchestra.run.smithi090.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 636 kB/s | 43 kB 00:00 2023-11-25T04:54:46.092 INFO:teuthology.orchestra.run.smithi090.stdout:(7/13): ceph-mgr-rook-18.0.0-7506.gec928368.el9 110 kB/s | 50 kB 00:00 2023-11-25T04:54:46.360 INFO:teuthology.orchestra.run.smithi090.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 48 kB/s | 15 kB 00:00 2023-11-25T04:54:46.373 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:54:46.401 INFO:teuthology.orchestra.run.smithi090.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 86 kB/s | 32 kB 00:00 2023-11-25T04:54:46.546 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-11-25T04:54:46.610 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-11-25T04:54:46.660 INFO:teuthology.orchestra.run.smithi090.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 228 kB/s | 59 kB 00:00 2023-11-25T04:54:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-11-25T04:54:46.710 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-11-25T04:54:46.727 INFO:teuthology.orchestra.run.smithi090.stdout:(11/13): python3-google-auth-2.23.4-1.el9.noarc 311 kB/s | 197 kB 00:00 2023-11-25T04:54:46.751 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-11-25T04:54:46.848 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-11-25T04:54:46.869 INFO:teuthology.orchestra.run.smithi090.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 430 kB/s | 90 kB 00:00 2023-11-25T04:54:46.976 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-11-25T04:54:47.030 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-11-25T04:54:47.036 INFO:teuthology.orchestra.run.smithi090.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2023-11-25T04:54:47.038 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:47.039 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2023-11-25T04:54:47.039 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:54:47.053 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:54:47.054 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:54:47.156 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:54:47.157 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:54:47.680 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:54:47.793 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-11-25T04:54:47.842 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-11-25T04:54:47.850 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-11-25T04:54:47.875 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-11-25T04:54:47.905 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-11-25T04:54:47.924 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-11-25T04:54:47.953 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-rook-2:18.0.0-7506.gec928368.el9.noarch 13/13 2023-11-25T04:54:47.968 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-11-25T04:54:48.007 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-11-25T04:54:48.048 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-11-25T04:54:48.141 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-11-25T04:54:48.252 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-11-25T04:54:48.308 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-11-25T04:54:48.766 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7506.gec928368.el9.noarch 13/13 2023-11-25T04:54:48.766 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7506.gec928368.el9.noarch 1/13 2023-11-25T04:54:48.766 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-11-25T04:54:48.766 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-11-25T04:54:48.766 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-11-25T04:54:48.766 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-11-25T04:54:48.766 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-11-25T04:54:48.766 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-11-25T04:54:48.766 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-11-25T04:54:48.767 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-11-25T04:54:48.767 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-11-25T04:54:48.767 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-11-25T04:54:48.767 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-11-25T04:54:49.034 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-11-25T04:54:49.100 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-11-25T04:54:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-11-25T04:54:49.137 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-11-25T04:54:49.137 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:49.137 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:54:49.137 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:49.137 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-25T04:54:49.137 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-25T04:54:49.137 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-11-25T04:54:49.137 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-25T04:54:49.137 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-25T04:54:49.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-25T04:54:49.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-25T04:54:49.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-25T04:54:49.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-25T04:54:49.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-25T04:54:49.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-25T04:54:49.138 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-25T04:54:49.138 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:49.138 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:54:49.190 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-11-25T04:54:49.229 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:18.0.0-7506.gec928368.el9.noarch 13/13 2023-11-25T04:54:49.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-cephadm 2023-11-25T04:54:50.007 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:20 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:54:50.158 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:54:50.159 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2023-11-25T04:54:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-25T04:54:50.159 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2023-11-25T04:54:50.159 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:54:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 133 k 2023-11-25T04:54:50.159 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-25T04:54:50.159 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-25T04:54:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-25T04:54:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-25T04:54:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-25T04:54:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-25T04:54:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:50.160 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:54:50.160 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2023-11-25T04:54:50.160 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2023-11-25T04:54:50.160 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:50.161 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 6.8 M 2023-11-25T04:54:50.161 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 33 M 2023-11-25T04:54:50.161 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:54:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7506.gec928368.el9.noarch 13/13 2023-11-25T04:54:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7506.gec928368.el9.noarch 1/13 2023-11-25T04:54:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-11-25T04:54:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-11-25T04:54:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-11-25T04:54:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-11-25T04:54:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-11-25T04:54:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-11-25T04:54:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-11-25T04:54:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-11-25T04:54:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-11-25T04:54:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-11-25T04:54:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-11-25T04:54:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-11-25T04:54:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:50.568 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:54:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-25T04:54:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-25T04:54:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-11-25T04:54:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-25T04:54:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-25T04:54:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-25T04:54:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-25T04:54:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-25T04:54:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-25T04:54:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-25T04:54:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-25T04:54:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-25T04:54:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:50.569 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:54:50.760 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 621 kB/s | 228 kB 00:00 2023-11-25T04:54:50.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-cephadm 2023-11-25T04:54:51.043 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): ceph-mgr-cephadm-18.0.0-7506.gec928368.e 204 kB/s | 133 kB 00:00 2023-11-25T04:54:51.119 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.0 MB/s | 5.8 MB 00:00 2023-11-25T04:54:51.441 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:22 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:54:51.460 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 139 kB/s | 58 kB 00:00 2023-11-25T04:54:51.511 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 730 kB/s | 548 kB 00:00 2023-11-25T04:54:51.536 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 122 kB/s | 51 kB 00:00 2023-11-25T04:54:51.538 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:51.538 INFO:teuthology.orchestra.run.smithi007.stdout:Total 4.9 MB/s | 6.8 MB 00:01 2023-11-25T04:54:51.539 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:54:51.549 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:54:51.550 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:54:51.584 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:54:51.585 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2023-11-25T04:54:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2023-11-25T04:54:51.585 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2023-11-25T04:54:51.586 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:54:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 133 k 2023-11-25T04:54:51.586 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2023-11-25T04:54:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-25T04:54:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-25T04:54:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-25T04:54:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-25T04:54:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-25T04:54:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:51.586 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:54:51.586 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2023-11-25T04:54:51.587 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2023-11-25T04:54:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:51.587 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 6.8 M 2023-11-25T04:54:51.587 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 33 M 2023-11-25T04:54:51.587 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:54:51.648 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:54:51.650 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:54:51.922 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): ceph-mgr-cephadm-18.0.0-7506.gec928368.e 469 kB/s | 133 kB 00:00 2023-11-25T04:54:51.947 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 742 kB/s | 228 kB 00:00 2023-11-25T04:54:51.950 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:54:52.128 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-11-25T04:54:52.164 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2023-11-25T04:54:52.181 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-11-25T04:54:52.189 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 239 kB/s | 58 kB 00:00 2023-11-25T04:54:52.348 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 278 kB/s | 51 kB 00:00 2023-11-25T04:54:52.532 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 900 kB/s | 548 kB 00:00 2023-11-25T04:54:52.533 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:52.534 INFO:teuthology.orchestra.run.smithi090.stdout:Total 7.1 MB/s | 6.8 MB 00:00 2023-11-25T04:54:52.534 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:54:52.543 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:54:52.543 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:54:52.631 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:54:52.631 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:54:52.727 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-11-25T04:54:52.827 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-11-25T04:54:52.895 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-11-25T04:54:52.925 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7506.gec928368.el9.noarch 6/6 2023-11-25T04:54:52.926 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:54:53.079 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-11-25T04:54:53.133 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-11-25T04:54:53.654 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-11-25T04:54:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7506.gec928368.el9.noarch 6/6 2023-11-25T04:54:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7506.gec928368.el9.noarch 1/6 2023-11-25T04:54:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-11-25T04:54:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-11-25T04:54:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-11-25T04:54:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-11-25T04:54:53.762 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-11-25T04:54:53.824 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-11-25T04:54:53.848 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7506.gec928368.el9.noarch 6/6 2023-11-25T04:54:54.074 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-11-25T04:54:54.074 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:54.074 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:54:54.075 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:54.075 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-25T04:54:54.075 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-25T04:54:54.075 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-25T04:54:54.075 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-25T04:54:54.075 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-25T04:54:54.075 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:54.075 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:54:54.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-fuse 2023-11-25T04:54:54.908 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7506.gec928368.el9.noarch 6/6 2023-11-25T04:54:54.908 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7506.gec928368.el9.noarch 1/6 2023-11-25T04:54:54.909 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-11-25T04:54:54.909 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-11-25T04:54:54.909 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-11-25T04:54:54.909 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-11-25T04:54:54.939 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:24 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:54:55.092 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:54:55.093 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:54:55.094 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-25T04:54:55.094 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:54:55.094 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:54:55.094 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse x86_64 2:18.0.0-7506.gec928368.el9 ceph 859 k 2023-11-25T04:54:55.094 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-25T04:54:55.095 INFO:teuthology.orchestra.run.smithi007.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-25T04:54:55.095 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:55.095 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:54:55.095 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:54:55.095 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2023-11-25T04:54:55.095 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:55.096 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 938 k 2023-11-25T04:54:55.096 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.7 M 2023-11-25T04:54:55.096 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:54:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-11-25T04:54:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:55.216 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:54:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-25T04:54:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-25T04:54:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-25T04:54:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-25T04:54:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-25T04:54:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:55.217 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:54:55.271 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 442 kB/s | 79 kB 00:00 2023-11-25T04:54:55.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-fuse 2023-11-25T04:54:55.547 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-fuse-18.0.0-7506.gec928368.el9.x86_ 1.9 MB/s | 859 kB 00:00 2023-11-25T04:54:55.548 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:55.548 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.0 MB/s | 938 kB 00:00 2023-11-25T04:54:55.548 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:54:55.570 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:54:55.570 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:54:55.650 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:54:55.650 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:54:55.844 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:54:55.967 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-11-25T04:54:56.032 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-2:18.0.0-7506.gec928368.el9.x86_64 2/2 2023-11-25T04:54:56.079 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:27 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:54:56.225 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:54:56.225 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:54:56.225 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2023-11-25T04:54:56.225 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:54:56.225 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:54:56.225 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:18.0.0-7506.gec928368.el9 ceph 859 k 2023-11-25T04:54:56.226 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2023-11-25T04:54:56.226 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-25T04:54:56.226 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:56.226 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:54:56.226 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:54:56.226 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2023-11-25T04:54:56.226 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:56.227 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 938 k 2023-11-25T04:54:56.227 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 2.7 M 2023-11-25T04:54:56.227 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:54:56.438 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 372 kB/s | 79 kB 00:00 2023-11-25T04:54:56.750 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7506.gec928368.el9.x86_64 2/2 2023-11-25T04:54:56.750 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-2:18.0.0-7506.gec928368.el9.x86_64 1/2 2023-11-25T04:54:57.066 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-11-25T04:54:57.066 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:57.067 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:54:57.067 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-2:18.0.0-7506.gec928368.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-11-25T04:54:57.067 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:57.067 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:54:57.263 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): ceph-fuse-18.0.0-7506.gec928368.el9.x86_ 829 kB/s | 859 kB 00:01 2023-11-25T04:54:57.264 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:57.265 INFO:teuthology.orchestra.run.smithi090.stdout:Total 903 kB/s | 938 kB 00:01 2023-11-25T04:54:57.265 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:54:57.285 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:54:57.285 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:54:57.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-volume 2023-11-25T04:54:57.360 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:54:57.361 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:54:57.540 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:54:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-11-25T04:54:57.762 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:18.0.0-7506.gec928368.el9.x86_64 2/2 2023-11-25T04:54:57.898 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:27 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:54:58.051 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:54:58.052 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:54:58.052 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-25T04:54:58.052 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:54:58.052 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:54:58.052 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 262 k 2023-11-25T04:54:58.052 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:58.053 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:54:58.053 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:54:58.053 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T04:54:58.053 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:58.053 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 262 k 2023-11-25T04:54:58.053 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.2 M 2023-11-25T04:54:58.053 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:54:58.285 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-volume-18.0.0-7506.gec928368.el9.noarch.rp 1.1 MB/s | 262 kB 00:00 2023-11-25T04:54:58.286 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:54:58.286 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.1 MB/s | 262 kB 00:00 2023-11-25T04:54:58.286 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:54:58.295 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:54:58.295 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:54:58.330 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:54:58.330 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:54:58.503 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7506.gec928368.el9.x86_64 2/2 2023-11-25T04:54:58.503 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:18.0.0-7506.gec928368.el9.x86_64 1/2 2023-11-25T04:54:58.560 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:54:58.621 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-volume-2:18.0.0-7506.gec928368.el9.noarch 1/1 2023-11-25T04:54:58.637 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-volume-2:18.0.0-7506.gec928368.el9.noarch 1/1 2023-11-25T04:54:58.638 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:54:58.638 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-25T04:54:58.638 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:58.842 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-11-25T04:54:58.842 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:58.843 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:54:58.843 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:18.0.0-7506.gec928368.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-11-25T04:54:58.843 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:58.843 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:54:59.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-volume 2023-11-25T04:54:59.691 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-volume-2:18.0.0-7506.gec928368.el9.noarch 1/1 2023-11-25T04:54:59.692 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:59.692 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:54:59.692 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:54:59.692 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:54:59.692 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:54:59.748 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:30 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:54:59.899 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:54:59.899 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:54:59.899 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2023-11-25T04:54:59.900 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:54:59.900 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:54:59.900 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume noarch 2:18.0.0-7506.gec928368.el9 ceph-noarch 262 k 2023-11-25T04:54:59.900 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:59.900 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:54:59.900 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:54:59.900 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T04:54:59.900 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:54:59.901 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 262 k 2023-11-25T04:54:59.901 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 1.2 M 2023-11-25T04:54:59.901 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:55:00.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados-devel 2023-11-25T04:55:00.148 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-volume-18.0.0-7506.gec928368.el9.noarch.rp 1.0 MB/s | 262 kB 00:00 2023-11-25T04:55:00.149 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:55:00.149 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.0 MB/s | 262 kB 00:00 2023-11-25T04:55:00.149 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:55:00.157 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:55:00.157 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:55:00.192 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:55:00.193 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:55:00.407 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:55:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-volume-2:18.0.0-7506.gec928368.el9.noarch 1/1 2023-11-25T04:55:00.482 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-volume-2:18.0.0-7506.gec928368.el9.noarch 1/1 2023-11-25T04:55:00.482 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:55:00.482 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-25T04:55:00.483 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:00.629 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:30 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:55:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:55:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:00.782 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T04:55:00.782 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:00.782 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:55:00.782 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel x86_64 2:18.0.0-7506.gec928368.el9 ceph 126 k 2023-11-25T04:55:00.782 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:00.782 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:55:00.782 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:00.782 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T04:55:00.782 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:00.783 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 126 k 2023-11-25T04:55:00.783 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 456 k 2023-11-25T04:55:00.783 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:55:01.091 INFO:teuthology.orchestra.run.smithi007.stdout:librados-devel-18.0.0-7506.gec928368.el9.x86_64 411 kB/s | 126 kB 00:00 2023-11-25T04:55:01.092 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:55:01.092 INFO:teuthology.orchestra.run.smithi007.stdout:Total 408 kB/s | 126 kB 00:00 2023-11-25T04:55:01.092 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:55:01.098 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:55:01.098 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:55:01.156 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:55:01.157 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:55:01.306 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:55:01.417 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:01.650 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-volume-2:18.0.0-7506.gec928368.el9.noarch 1/1 2023-11-25T04:55:01.650 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:01.650 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:55:01.651 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume-2:18.0.0-7506.gec928368.el9.noarch 2023-11-25T04:55:01.651 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:01.651 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:55:01.894 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados-devel-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:02.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados-devel 2023-11-25T04:55:02.231 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:02.231 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:02.231 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:55:02.231 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:55:02.231 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:02.231 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:55:02.495 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs2 2023-11-25T04:55:02.644 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:33 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:55:02.789 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:55:02.790 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T04:55:02.790 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:02.790 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:55:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:18.0.0-7506.gec928368.el9 ceph 126 k 2023-11-25T04:55:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:02.790 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:55:02.790 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:02.790 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T04:55:02.791 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:02.791 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 126 k 2023-11-25T04:55:02.791 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 456 k 2023-11-25T04:55:02.791 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:55:02.993 INFO:teuthology.orchestra.run.smithi090.stdout:librados-devel-18.0.0-7506.gec928368.el9.x86_64 628 kB/s | 126 kB 00:00 2023-11-25T04:55:02.993 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:55:02.993 INFO:teuthology.orchestra.run.smithi090.stdout:Total 623 kB/s | 126 kB 00:00 2023-11-25T04:55:02.993 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:55:02.999 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:55:02.999 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:55:03.054 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:55:03.054 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:55:03.102 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:33 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:55:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:55:03.201 INFO:teuthology.orchestra.run.smithi007.stdout:Package libcephfs2-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:55:03.254 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:55:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-25T04:55:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:55:03.291 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:03.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs-devel 2023-11-25T04:55:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados-devel-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:03.936 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:33 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:55:04.089 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:55:04.089 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:04.089 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T04:55:04.090 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:04.090 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:55:04.090 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel x86_64 2:18.0.0-7506.gec928368.el9 ceph 31 k 2023-11-25T04:55:04.090 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:04.090 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:55:04.090 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:04.090 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T04:55:04.090 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:04.090 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 31 k 2023-11-25T04:55:04.090 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 139 k 2023-11-25T04:55:04.090 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:55:04.156 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:04.156 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:04.156 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:55:04.156 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:55:04.156 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:04.156 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:55:04.254 INFO:teuthology.orchestra.run.smithi007.stdout:libcephfs-devel-18.0.0-7506.gec928368.el9.x86_6 190 kB/s | 31 kB 00:00 2023-11-25T04:55:04.254 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:55:04.254 INFO:teuthology.orchestra.run.smithi007.stdout:Total 189 kB/s | 31 kB 00:00 2023-11-25T04:55:04.255 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:55:04.258 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:55:04.258 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:55:04.288 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:55:04.289 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:55:04.386 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:55:04.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs2 2023-11-25T04:55:04.513 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs-devel-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:04.916 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:05.032 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:36 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:55:05.132 INFO:teuthology.orchestra.run.smithi090.stdout:Package libcephfs2-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:55:05.185 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:55:05.186 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2023-11-25T04:55:05.186 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:55:05.207 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs-devel-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:05.207 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:05.207 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:55:05.207 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:55:05.207 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:05.208 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:55:05.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs-devel 2023-11-25T04:55:05.399 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados2 2023-11-25T04:55:05.827 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:36 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:55:05.970 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:55:05.971 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:05.971 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T04:55:05.971 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:05.971 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:55:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:18.0.0-7506.gec928368.el9 ceph 31 k 2023-11-25T04:55:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:05.972 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:55:05.972 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:05.972 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T04:55:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:05.973 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 31 k 2023-11-25T04:55:05.973 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 139 k 2023-11-25T04:55:05.973 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:55:06.007 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:36 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:55:06.105 INFO:teuthology.orchestra.run.smithi007.stdout:Package librados2-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:55:06.159 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:55:06.160 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-25T04:55:06.160 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:55:06.175 INFO:teuthology.orchestra.run.smithi090.stdout:libcephfs-devel-18.0.0-7506.gec928368.el9.x86_6 153 kB/s | 31 kB 00:00 2023-11-25T04:55:06.176 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:55:06.176 INFO:teuthology.orchestra.run.smithi090.stdout:Total 152 kB/s | 31 kB 00:00 2023-11-25T04:55:06.176 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:55:06.179 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:55:06.179 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:55:06.214 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:55:06.214 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:55:06.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librbd1 2023-11-25T04:55:06.303 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:55:06.422 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:06.830 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:36 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:55:06.870 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:06.928 INFO:teuthology.orchestra.run.smithi007.stdout:Package librbd1-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:55:06.982 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:55:06.983 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-25T04:55:06.983 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:55:07.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rados 2023-11-25T04:55:07.127 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:07.127 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:07.127 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:55:07.127 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:55:07.127 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:07.127 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:55:07.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados2 2023-11-25T04:55:07.643 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:37 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:55:07.742 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rados-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:55:07.797 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:55:07.798 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-25T04:55:07.798 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:55:07.865 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rgw 2023-11-25T04:55:07.893 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:38 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:55:07.986 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:55:08.037 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:55:08.038 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2023-11-25T04:55:08.038 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:55:08.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librbd1 2023-11-25T04:55:08.466 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:38 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:55:08.565 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rgw-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:55:08.617 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:55:08.618 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-25T04:55:08.618 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:55:08.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-cephfs 2023-11-25T04:55:08.706 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:39 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:55:08.801 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:55:08.852 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:55:08.853 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2023-11-25T04:55:08.853 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:55:08.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rados 2023-11-25T04:55:09.292 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:39 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:55:09.391 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-cephfs-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:55:09.444 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:55:09.445 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-25T04:55:09.445 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:55:09.506 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rbd 2023-11-25T04:55:09.513 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:40 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:55:09.609 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rados-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:55:09.659 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:55:09.660 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2023-11-25T04:55:09.660 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:55:09.779 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rgw 2023-11-25T04:55:10.108 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:40 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:55:10.206 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rbd-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:55:10.260 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:55:10.261 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-25T04:55:10.261 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:55:10.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-fuse 2023-11-25T04:55:10.352 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:41 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:55:10.447 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rgw-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:55:10.498 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:55:10.499 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2023-11-25T04:55:10.499 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:55:10.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-cephfs 2023-11-25T04:55:10.942 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:40 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:55:11.094 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:55:11.095 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:11.095 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2023-11-25T04:55:11.095 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:11.095 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:55:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse x86_64 2:18.0.0-7506.gec928368.el9 ceph 85 k 2023-11-25T04:55:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:11.096 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:55:11.096 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:11.096 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T04:55:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:11.096 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 85 k 2023-11-25T04:55:11.096 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 231 k 2023-11-25T04:55:11.096 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:55:11.150 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:42 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:55:11.243 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-cephfs-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:55:11.293 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:55:11.294 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2023-11-25T04:55:11.294 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:55:11.307 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-fuse-18.0.0-7506.gec928368.el9.x86_64.rpm 403 kB/s | 85 kB 00:00 2023-11-25T04:55:11.307 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:55:11.308 INFO:teuthology.orchestra.run.smithi007.stdout:Total 401 kB/s | 85 kB 00:00 2023-11-25T04:55:11.308 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:55:11.315 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:55:11.316 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:55:11.373 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:55:11.373 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:55:11.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rbd 2023-11-25T04:55:11.515 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:55:11.616 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:11.993 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:42 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:55:12.029 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:12.089 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rbd-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:55:12.138 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:55:12.139 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2023-11-25T04:55:12.140 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:55:12.241 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-fuse 2023-11-25T04:55:12.365 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:12.366 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:12.366 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:55:12.366 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:55:12.366 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:12.366 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:55:12.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-mirror 2023-11-25T04:55:12.813 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:43 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:55:12.957 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:55:12.958 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:12.958 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2023-11-25T04:55:12.958 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:12.958 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:55:12.959 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:18.0.0-7506.gec928368.el9 ceph 85 k 2023-11-25T04:55:12.959 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:12.959 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:55:12.959 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:12.959 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T04:55:12.959 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:12.959 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 85 k 2023-11-25T04:55:12.959 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 231 k 2023-11-25T04:55:12.959 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:55:13.212 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:43 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:55:13.365 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:55:13.366 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:13.366 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T04:55:13.366 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:13.366 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:55:13.366 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror x86_64 2:18.0.0-7506.gec928368.el9 ceph 3.0 M 2023-11-25T04:55:13.366 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:13.366 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:55:13.366 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:13.367 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T04:55:13.367 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:13.367 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 3.0 M 2023-11-25T04:55:13.367 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 12 M 2023-11-25T04:55:13.367 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:55:13.439 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-fuse-18.0.0-7506.gec928368.el9.x86_64.rpm 177 kB/s | 85 kB 00:00 2023-11-25T04:55:13.440 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:55:13.440 INFO:teuthology.orchestra.run.smithi090.stdout:Total 177 kB/s | 85 kB 00:00 2023-11-25T04:55:13.440 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:55:13.447 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:55:13.447 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:55:13.502 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:55:13.502 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:55:13.640 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:55:13.746 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:14.002 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-mirror-18.0.0-7506.gec928368.el9.x86_64.rpm 4.7 MB/s | 3.0 MB 00:00 2023-11-25T04:55:14.003 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:55:14.003 INFO:teuthology.orchestra.run.smithi007.stdout:Total 4.7 MB/s | 3.0 MB 00:00 2023-11-25T04:55:14.003 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:55:14.014 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:55:14.014 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:55:14.092 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:55:14.093 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:55:14.185 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:14.454 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:14.454 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:14.454 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:55:14.454 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:55:14.454 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:14.454 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:55:14.522 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:55:14.586 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:14.612 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:14.613 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:55:14.613 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-25T04:55:14.613 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-25T04:55:14.613 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-25T04:55:14.613 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:14.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-mirror 2023-11-25T04:55:15.300 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:46 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:55:15.450 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:55:15.450 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:15.450 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T04:55:15.451 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:15.451 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:55:15.451 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:18.0.0-7506.gec928368.el9 ceph 3.0 M 2023-11-25T04:55:15.451 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:15.451 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:55:15.451 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:15.451 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T04:55:15.451 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:15.451 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 3.0 M 2023-11-25T04:55:15.451 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 12 M 2023-11-25T04:55:15.451 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:55:15.738 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:15.739 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:15.739 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:55:15.739 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:55:15.739 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:15.739 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:55:16.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-nbd 2023-11-25T04:55:16.621 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:46 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:55:16.774 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:55:16.775 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:16.775 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2023-11-25T04:55:16.775 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:16.775 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:55:16.775 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd x86_64 2:18.0.0-7506.gec928368.el9 ceph 170 k 2023-11-25T04:55:16.775 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:16.776 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:55:16.776 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:16.776 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T04:55:16.776 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:16.776 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 170 k 2023-11-25T04:55:16.776 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 490 k 2023-11-25T04:55:16.776 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:55:17.431 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-mirror-18.0.0-7506.gec928368.el9.x86_64.rpm 1.5 MB/s | 3.0 MB 00:01 2023-11-25T04:55:17.431 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:55:17.432 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.5 MB/s | 3.0 MB 00:01 2023-11-25T04:55:17.432 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:55:17.443 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:55:17.443 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:55:17.481 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-nbd-18.0.0-7506.gec928368.el9.x86_64.rpm 241 kB/s | 170 kB 00:00 2023-11-25T04:55:17.481 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:55:17.482 INFO:teuthology.orchestra.run.smithi007.stdout:Total 240 kB/s | 170 kB 00:00 2023-11-25T04:55:17.482 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:55:17.491 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:55:17.491 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:55:17.518 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:55:17.519 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:55:17.558 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:55:17.558 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:55:17.712 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:55:17.807 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:17.926 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:55:17.990 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:18.014 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:18.014 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-25T04:55:18.014 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-25T04:55:18.014 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-25T04:55:18.015 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-25T04:55:18.015 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:18.281 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:18.623 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:18.624 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:18.624 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:55:18.624 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:55:18.624 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:18.624 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:55:18.820 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base-debuginfo 2023-11-25T04:55:19.257 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:19.257 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:19.257 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:55:19.257 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:55:19.257 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:19.258 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:55:19.446 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:49 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:55:19.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-nbd 2023-11-25T04:55:19.598 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:55:19.599 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:19.599 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T04:55:19.599 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:19.599 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:55:19.599 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 73 M 2023-11-25T04:55:19.599 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-25T04:55:19.599 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 21 M 2023-11-25T04:55:19.600 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:19.600 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:55:19.600 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:19.600 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2023-11-25T04:55:19.600 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:19.600 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 94 M 2023-11-25T04:55:19.600 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 508 M 2023-11-25T04:55:19.601 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:55:20.132 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:51 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:55:20.285 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:55:20.285 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2023-11-25T04:55:20.286 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:20.286 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:55:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:18.0.0-7506.gec928368.el9 ceph 170 k 2023-11-25T04:55:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:20.286 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:55:20.286 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:20.286 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T04:55:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:20.287 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 170 k 2023-11-25T04:55:20.287 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 490 k 2023-11-25T04:55:20.287 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:55:21.188 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-nbd-18.0.0-7506.gec928368.el9.x86_64.rpm 188 kB/s | 170 kB 00:00 2023-11-25T04:55:21.188 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:55:21.189 INFO:teuthology.orchestra.run.smithi090.stdout:Total 188 kB/s | 170 kB 00:00 2023-11-25T04:55:21.189 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:55:21.197 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:55:21.198 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:55:21.260 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:55:21.260 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:55:21.403 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:55:21.513 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:22.049 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:22.335 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T04:55:22.335 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:22.336 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:55:22.336 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:55:22.336 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:22.336 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:55:22.643 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-base-debuginfo 2023-11-25T04:55:23.077 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): ceph-base-debuginfo-18.0.0-7506.gec92836 21 MB/s | 73 MB 00:03 2023-11-25T04:55:23.235 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:54 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:55:23.386 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:55:23.391 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:23.392 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T04:55:23.392 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:23.392 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:55:23.392 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 73 M 2023-11-25T04:55:23.392 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2023-11-25T04:55:23.392 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 21 M 2023-11-25T04:55:23.392 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:23.392 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:55:23.393 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:23.393 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2023-11-25T04:55:23.393 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:23.393 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 94 M 2023-11-25T04:55:23.393 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 508 M 2023-11-25T04:55:23.393 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:55:25.222 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): ceph-debuginfo-18.0.0-7506.gec928368.el9 11 MB/s | 21 MB 00:01 2023-11-25T04:55:30.897 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): ceph-base-debuginfo-18.0.0-7506.gec92836 9.7 MB/s | 73 MB 00:07 2023-11-25T04:55:30.899 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:55:30.899 INFO:teuthology.orchestra.run.smithi090.stdout:Total 12 MB/s | 94 MB 00:07 2023-11-25T04:55:30.900 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:55:30.904 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:55:30.905 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:55:31.512 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:55:31.512 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:55:31.542 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-debuginfo-18.0.0-7506.gec928368.el9 1.7 MB/s | 21 MB 00:11 2023-11-25T04:55:31.544 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:55:31.545 INFO:teuthology.orchestra.run.smithi007.stdout:Total 7.8 MB/s | 94 MB 00:11 2023-11-25T04:55:31.545 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:55:31.550 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:55:31.551 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:55:32.193 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:55:32.193 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:55:34.462 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:55:35.296 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:55:42.048 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 1/2 2023-11-25T04:55:42.151 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 2/2 2023-11-25T04:55:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 1/2 2023-11-25T04:55:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 2/2 2023-11-25T04:55:46.610 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 2/2 2023-11-25T04:55:46.610 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/2 2023-11-25T04:55:46.888 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2/2 2023-11-25T04:55:46.888 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:46.888 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:55:46.888 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:55:46.888 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:55:46.888 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:46.888 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:55:47.030 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 2/2 2023-11-25T04:55:47.030 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/2 2023-11-25T04:55:47.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-common-debuginfo 2023-11-25T04:55:47.325 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2/2 2023-11-25T04:55:47.325 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:47.325 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:55:47.325 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:55:47.325 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:55:47.325 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:47.325 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:55:47.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-common-debuginfo 2023-11-25T04:55:47.724 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:18 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:55:47.869 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:55:47.870 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:47.870 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T04:55:47.870 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:47.871 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:55:47.871 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 405 M 2023-11-25T04:55:47.871 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:47.871 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:55:47.871 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:55:47.871 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T04:55:47.871 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:55:47.871 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 405 M 2023-11-25T04:55:47.871 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 1.6 G 2023-11-25T04:55:47.871 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:55:48.161 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:18 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:55:48.313 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:55:48.314 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:48.314 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T04:55:48.315 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:48.315 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:55:48.315 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 405 M 2023-11-25T04:55:48.315 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:48.315 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:55:48.315 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:55:48.315 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T04:55:48.315 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:55:48.315 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 405 M 2023-11-25T04:55:48.315 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.6 G 2023-11-25T04:55:48.315 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:56:16.044 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-common-debuginfo-18.0.0-7506.gec928368.el9 15 MB/s | 405 MB 00:27 2023-11-25T04:56:16.045 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:56:16.045 INFO:teuthology.orchestra.run.smithi007.stdout:Total 15 MB/s | 405 MB 00:27 2023-11-25T04:56:16.046 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:56:16.050 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:56:16.050 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:56:18.578 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:56:18.578 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:56:20.990 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-common-debuginfo-18.0.0-7506.gec928368.el9 12 MB/s | 405 MB 00:33 2023-11-25T04:56:20.991 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:56:20.991 INFO:teuthology.orchestra.run.smithi090.stdout:Total 12 MB/s | 405 MB 00:33 2023-11-25T04:56:20.992 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:56:20.996 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:56:20.996 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:56:23.401 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:56:23.401 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:57:01.877 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:57:01.974 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T04:57:04.399 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:57:04.499 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T04:57:09.828 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T04:57:10.103 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T04:57:10.103 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:57:10.103 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:57:10.103 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:57:10.103 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:57:10.103 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:57:10.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-11-25T04:57:10.949 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:40 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:57:11.099 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:57:11.099 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2023-11-25T04:57:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T04:57:11.100 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2023-11-25T04:57:11.100 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:57:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 1.4 M 2023-11-25T04:57:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:57:11.100 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:57:11.100 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2023-11-25T04:57:11.100 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T04:57:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:57:11.101 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 1.4 M 2023-11-25T04:57:11.101 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 6.2 M 2023-11-25T04:57:11.101 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:57:11.534 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-75 3.2 MB/s | 1.4 MB 00:00 2023-11-25T04:57:11.535 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:57:11.535 INFO:teuthology.orchestra.run.smithi007.stdout:Total 3.2 MB/s | 1.4 MB 00:00 2023-11-25T04:57:11.535 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:57:11.539 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:57:11.539 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:57:11.584 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:57:11.584 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:57:11.859 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:57:11.961 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-7506. 1/1 2023-11-25T04:57:12.384 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-7506. 1/1 2023-11-25T04:57:12.670 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-7506. 1/1 2023-11-25T04:57:12.671 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:57:12.671 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:57:12.671 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:57:12.671 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:57:12.671 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:57:12.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw-debuginfo 2023-11-25T04:57:13.484 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:43 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:57:13.637 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:57:13.638 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:57:13.638 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T04:57:13.638 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:57:13.638 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:57:13.638 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 173 M 2023-11-25T04:57:13.638 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:57:13.638 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:57:13.638 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:57:13.638 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T04:57:13.638 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:57:13.638 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 173 M 2023-11-25T04:57:13.639 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 732 M 2023-11-25T04:57:13.639 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:57:15.700 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T04:57:15.977 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T04:57:15.977 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:57:15.977 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:57:15.977 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:57:15.977 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:57:15.977 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:57:16.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-11-25T04:57:16.844 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:03:47 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:57:17.000 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:57:17.000 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2023-11-25T04:57:17.001 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T04:57:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2023-11-25T04:57:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:57:17.001 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 1.4 M 2023-11-25T04:57:17.001 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:57:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:57:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2023-11-25T04:57:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T04:57:17.001 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:57:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 1.4 M 2023-11-25T04:57:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 6.2 M 2023-11-25T04:57:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:57:18.074 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-75 1.3 MB/s | 1.4 MB 00:01 2023-11-25T04:57:18.074 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:57:18.075 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.3 MB/s | 1.4 MB 00:01 2023-11-25T04:57:18.075 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:57:18.079 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:57:18.079 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:57:18.119 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:57:18.120 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:57:18.366 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:57:18.472 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-7506. 1/1 2023-11-25T04:57:18.972 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-7506. 1/1 2023-11-25T04:57:19.244 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-7506. 1/1 2023-11-25T04:57:19.244 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:57:19.244 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:57:19.244 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:57:19.244 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:57:19.244 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:57:19.449 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw-debuginfo 2023-11-25T04:57:20.029 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:03:51 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:57:20.175 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:57:20.176 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:57:20.176 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T04:57:20.176 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:57:20.176 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:57:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 173 M 2023-11-25T04:57:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:57:20.177 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:57:20.177 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:57:20.177 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T04:57:20.177 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:57:20.178 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 173 M 2023-11-25T04:57:20.178 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 732 M 2023-11-25T04:57:20.178 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:57:21.337 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-radosgw-debuginfo-18.0.0-7506.gec928368.el 22 MB/s | 173 MB 00:07 2023-11-25T04:57:21.337 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:57:21.338 INFO:teuthology.orchestra.run.smithi007.stdout:Total 22 MB/s | 173 MB 00:07 2023-11-25T04:57:21.338 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:57:21.342 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:57:21.342 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:57:22.438 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:57:22.439 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:57:31.311 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-radosgw-debuginfo-18.0.0-7506.gec928368.el 16 MB/s | 173 MB 00:11 2023-11-25T04:57:31.312 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:57:31.313 INFO:teuthology.orchestra.run.smithi090.stdout:Total 16 MB/s | 173 MB 00:11 2023-11-25T04:57:31.313 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:57:31.317 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:57:31.318 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:57:32.369 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:57:32.369 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:57:41.193 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:57:41.284 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-7506.gec928368.el9.x 1/1 2023-11-25T04:57:46.053 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-7506.gec928368.el9.x 1/1 2023-11-25T04:57:46.337 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-7506.gec928368.el9.x 1/1 2023-11-25T04:57:46.337 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:57:46.337 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:57:46.337 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:57:46.337 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:57:46.337 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:57:46.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-test-debuginfo 2023-11-25T04:57:47.133 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:17 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:57:47.282 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:57:47.283 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:57:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T04:57:47.283 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:57:47.283 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:57:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 623 M 2023-11-25T04:57:47.283 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:57:47.283 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:57:47.283 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:57:47.284 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T04:57:47.284 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:57:47.284 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 623 M 2023-11-25T04:57:47.284 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.8 G 2023-11-25T04:57:47.284 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:57:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:57:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-7506.gec928368.el9.x 1/1 2023-11-25T04:57:56.236 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-7506.gec928368.el9.x 1/1 2023-11-25T04:57:56.526 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-7506.gec928368.el9.x 1/1 2023-11-25T04:57:56.526 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:57:56.526 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:57:56.526 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:57:56.526 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:57:56.526 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:57:56.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-test-debuginfo 2023-11-25T04:57:57.335 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:04:28 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:57:57.478 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:57:57.479 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:57:57.479 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T04:57:57.479 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:57:57.479 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:57:57.479 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 623 M 2023-11-25T04:57:57.479 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:57:57.479 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:57:57.479 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:57:57.479 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T04:57:57.480 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:57:57.480 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 623 M 2023-11-25T04:57:57.480 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 2.8 G 2023-11-25T04:57:57.480 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:58:19.041 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-test-debuginfo-18.0.0-7506.gec928368.el9.x 20 MB/s | 623 MB 00:31 2023-11-25T04:58:19.041 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:58:19.042 INFO:teuthology.orchestra.run.smithi007.stdout:Total 20 MB/s | 623 MB 00:31 2023-11-25T04:58:19.042 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:58:19.048 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:58:19.048 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:58:22.911 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:58:22.912 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:58:36.650 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-test-debuginfo-18.0.0-7506.gec928368.el9.x 16 MB/s | 623 MB 00:39 2023-11-25T04:58:36.651 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:58:36.651 INFO:teuthology.orchestra.run.smithi090.stdout:Total 16 MB/s | 623 MB 00:39 2023-11-25T04:58:36.652 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T04:58:36.657 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T04:58:36.657 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T04:58:40.331 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T04:58:40.332 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T04:59:32.016 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:59:32.528 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T04:59:41.526 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T04:59:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T04:59:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:59:41.852 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:59:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:59:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:59:41.853 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:59:42.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base-debuginfo 2023-11-25T04:59:42.744 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:12 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:59:42.843 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-base-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:59:42.894 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:59:42.895 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-25T04:59:42.895 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:59:42.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-debuginfo 2023-11-25T04:59:43.571 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:13 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:59:43.721 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:59:43.722 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:59:43.722 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T04:59:43.722 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:59:43.722 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:59:43.722 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 19 M 2023-11-25T04:59:43.722 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:59:43.722 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:59:43.722 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:59:43.722 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T04:59:43.723 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:59:43.723 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 19 M 2023-11-25T04:59:43.723 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 100 M 2023-11-25T04:59:43.723 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:59:44.838 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-mgr-debuginfo-18.0.0-7506.gec928368.el9.x8 17 MB/s | 19 MB 00:01 2023-11-25T04:59:44.838 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:59:44.838 INFO:teuthology.orchestra.run.smithi007.stdout:Total 17 MB/s | 19 MB 00:01 2023-11-25T04:59:44.838 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:59:44.841 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:59:44.841 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:59:44.995 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:59:44.995 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:59:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T04:59:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T04:59:47.298 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:59:47.402 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T04:59:48.480 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T04:59:48.766 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T04:59:48.766 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:59:48.766 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:59:48.766 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:59:48.766 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:59:48.766 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:59:48.925 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mds-debuginfo 2023-11-25T04:59:49.537 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:19 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:59:49.687 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:59:49.688 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:59:49.688 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T04:59:49.688 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:59:49.688 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:59:49.688 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 47 M 2023-11-25T04:59:49.688 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:59:49.688 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:59:49.688 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:59:49.688 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T04:59:49.688 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:59:49.689 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 47 M 2023-11-25T04:59:49.689 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 168 M 2023-11-25T04:59:49.689 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T04:59:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-mds-debuginfo-18.0.0-7506.gec928368.el9.x8 21 MB/s | 47 MB 00:02 2023-11-25T04:59:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T04:59:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:Total 21 MB/s | 47 MB 00:02 2023-11-25T04:59:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T04:59:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T04:59:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T04:59:52.258 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T04:59:52.258 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T04:59:56.841 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T04:59:57.117 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T04:59:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:59:57.118 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T04:59:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:59:57.118 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:59:57.118 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:59:57.139 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T04:59:57.238 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T04:59:57.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-base-debuginfo 2023-11-25T04:59:57.999 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:28 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:59:58.094 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-base-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 is already installed. 2023-11-25T04:59:58.144 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:59:58.145 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2023-11-25T04:59:58.145 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T04:59:58.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-debuginfo 2023-11-25T04:59:58.710 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T04:59:58.803 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:29 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T04:59:58.945 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T04:59:58.946 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:59:58.946 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T04:59:58.946 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:59:58.946 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T04:59:58.946 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 19 M 2023-11-25T04:59:58.946 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:59:58.946 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T04:59:58.946 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T04:59:58.946 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T04:59:58.946 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T04:59:58.946 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 19 M 2023-11-25T04:59:58.949 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 100 M 2023-11-25T04:59:58.949 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T04:59:58.991 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T04:59:58.991 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:59:58.991 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T04:59:58.991 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T04:59:58.991 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:59:58.991 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T04:59:59.189 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mon-debuginfo 2023-11-25T04:59:59.798 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:29 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T04:59:59.948 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T04:59:59.948 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:59:59.948 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T04:59:59.948 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:59:59.948 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T04:59:59.948 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 111 M 2023-11-25T04:59:59.948 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:59:59.949 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T04:59:59.949 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T04:59:59.949 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T04:59:59.949 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T04:59:59.949 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 111 M 2023-11-25T04:59:59.949 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 507 M 2023-11-25T04:59:59.949 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T05:00:00.176 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-mgr-debuginfo-18.0.0-7506.gec928368.el9.x8 15 MB/s | 19 MB 00:01 2023-11-25T05:00:00.177 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:00:00.177 INFO:teuthology.orchestra.run.smithi090.stdout:Total 15 MB/s | 19 MB 00:01 2023-11-25T05:00:00.178 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:00:00.181 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:00:00.181 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:00:00.327 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:00:00.327 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:00:02.489 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:00:02.599 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:00:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:00:04.152 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:00:04.152 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:00:04.152 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:00:04.153 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:00:04.153 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:00:04.153 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:00:04.363 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mds-debuginfo 2023-11-25T05:00:04.941 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:35 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T05:00:05.084 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T05:00:05.085 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:00:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T05:00:05.085 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:00:05.085 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T05:00:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 47 M 2023-11-25T05:00:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:00:05.085 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T05:00:05.085 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:00:05.085 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T05:00:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:00:05.086 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 47 M 2023-11-25T05:00:05.086 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 168 M 2023-11-25T05:00:05.086 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T05:00:05.689 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-mon-debuginfo-18.0.0-7506.gec928368.el9.x8 19 MB/s | 111 MB 00:05 2023-11-25T05:00:05.690 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:00:05.691 INFO:teuthology.orchestra.run.smithi007.stdout:Total 19 MB/s | 111 MB 00:05 2023-11-25T05:00:05.691 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T05:00:05.695 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T05:00:05.696 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T05:00:06.413 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T05:00:06.414 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T05:00:07.423 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-mds-debuginfo-18.0.0-7506.gec928368.el9.x8 20 MB/s | 47 MB 00:02 2023-11-25T05:00:07.424 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:00:07.428 INFO:teuthology.orchestra.run.smithi090.stdout:Total 20 MB/s | 47 MB 00:02 2023-11-25T05:00:07.430 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:00:07.431 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:00:07.431 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:00:07.734 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:00:07.735 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:00:12.321 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:00:12.421 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:00:14.069 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:00:14.342 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:00:14.342 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:00:14.342 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:00:14.343 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:00:14.343 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:00:14.343 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:00:14.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mon-debuginfo 2023-11-25T05:00:15.151 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:46 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T05:00:15.294 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T05:00:15.295 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:00:15.295 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T05:00:15.295 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:00:15.295 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T05:00:15.296 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 111 M 2023-11-25T05:00:15.296 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:00:15.296 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T05:00:15.296 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:00:15.296 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T05:00:15.296 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:00:15.296 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 111 M 2023-11-25T05:00:15.296 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 507 M 2023-11-25T05:00:15.296 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T05:00:18.717 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T05:00:18.822 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:00:20.744 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-mon-debuginfo-18.0.0-7506.gec928368.el9.x8 20 MB/s | 111 MB 00:05 2023-11-25T05:00:20.745 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:00:20.745 INFO:teuthology.orchestra.run.smithi090.stdout:Total 20 MB/s | 111 MB 00:05 2023-11-25T05:00:20.745 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:00:20.749 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:00:20.749 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:00:21.435 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:00:21.435 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:00:22.132 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:00:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:00:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:00:22.422 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T05:00:22.422 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:00:22.422 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:00:22.422 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T05:00:22.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-osd-debuginfo 2023-11-25T05:00:23.223 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:53 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T05:00:23.374 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T05:00:23.375 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:00:23.375 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T05:00:23.375 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:00:23.375 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T05:00:23.375 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 311 M 2023-11-25T05:00:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:00:23.376 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T05:00:23.376 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:00:23.376 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T05:00:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:00:23.376 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 311 M 2023-11-25T05:00:23.376 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.3 G 2023-11-25T05:00:23.376 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T05:00:33.105 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:00:33.200 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:00:37.114 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-osd-debuginfo-18.0.0-7506.gec928368.el9.x8 23 MB/s | 311 MB 00:13 2023-11-25T05:00:37.115 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:00:37.116 INFO:teuthology.orchestra.run.smithi007.stdout:Total 23 MB/s | 311 MB 00:13 2023-11-25T05:00:37.116 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T05:00:37.120 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T05:00:37.120 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T05:00:37.397 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:00:37.659 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:00:37.659 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:00:37.661 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:00:37.661 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:00:37.661 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:00:37.661 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:00:37.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-osd-debuginfo 2023-11-25T05:00:38.499 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:07:09 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T05:00:38.641 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T05:00:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:00:38.642 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T05:00:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:00:38.643 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T05:00:38.643 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 311 M 2023-11-25T05:00:38.643 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:00:38.643 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T05:00:38.643 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:00:38.643 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T05:00:38.643 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:00:38.643 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 311 M 2023-11-25T05:00:38.643 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 1.3 G 2023-11-25T05:00:38.643 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T05:00:39.065 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T05:00:39.065 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T05:00:52.849 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-osd-debuginfo-18.0.0-7506.gec928368.el9.x8 22 MB/s | 311 MB 00:14 2023-11-25T05:00:52.850 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:00:52.850 INFO:teuthology.orchestra.run.smithi090.stdout:Total 22 MB/s | 311 MB 00:14 2023-11-25T05:00:52.851 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:00:52.854 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:00:52.854 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:00:54.709 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:00:54.709 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:01:12.869 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T05:01:12.961 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:01:21.296 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:01:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:01:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:21.605 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T05:01:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:01:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:21.606 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T05:01:21.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-fuse-debuginfo 2023-11-25T05:01:22.419 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:07:52 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T05:01:22.570 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T05:01:22.571 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:22.571 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T05:01:22.571 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:22.571 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T05:01:22.572 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 5.8 M 2023-11-25T05:01:22.572 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:22.572 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T05:01:22.572 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:22.572 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T05:01:22.572 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:22.573 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 5.8 M 2023-11-25T05:01:22.573 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 28 M 2023-11-25T05:01:22.573 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T05:01:23.063 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-fuse-debuginfo-18.0.0-7506.gec928368.el9.x 12 MB/s | 5.8 MB 00:00 2023-11-25T05:01:23.064 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:01:23.064 INFO:teuthology.orchestra.run.smithi007.stdout:Total 12 MB/s | 5.8 MB 00:00 2023-11-25T05:01:23.065 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T05:01:23.068 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T05:01:23.068 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T05:01:23.140 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T05:01:23.140 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T05:01:23.903 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T05:01:24.015 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T05:01:24.597 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T05:01:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T05:01:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:24.892 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T05:01:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:01:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:24.892 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T05:01:25.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados-devel-debuginfo 2023-11-25T05:01:25.666 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:07:55 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T05:01:25.817 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T05:01:25.817 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:25.817 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T05:01:25.817 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:25.818 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T05:01:25.818 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 108 k 2023-11-25T05:01:25.818 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:25.818 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T05:01:25.818 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:25.818 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T05:01:25.818 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:25.819 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 108 k 2023-11-25T05:01:25.819 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 440 k 2023-11-25T05:01:25.819 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T05:01:26.136 INFO:teuthology.orchestra.run.smithi007.stdout:librados-devel-debuginfo-18.0.0-7506.gec928368. 345 kB/s | 108 kB 00:00 2023-11-25T05:01:26.136 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:01:26.136 INFO:teuthology.orchestra.run.smithi007.stdout:Total 344 kB/s | 108 kB 00:00 2023-11-25T05:01:26.136 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T05:01:26.138 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T05:01:26.138 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T05:01:26.175 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T05:01:26.176 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T05:01:26.289 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T05:01:26.434 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-debuginfo-2:18.0.0-7506.gec928368.el9 1/1 2023-11-25T05:01:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:01:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:01:26.858 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-7506.gec928368.el9 1/1 2023-11-25T05:01:27.149 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-7506.gec928368.el9 1/1 2023-11-25T05:01:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:27.150 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T05:01:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:01:27.150 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:27.150 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T05:01:27.330 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs2-debuginfo 2023-11-25T05:01:27.934 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:07:57 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T05:01:28.083 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T05:01:28.084 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:28.084 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T05:01:28.084 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:28.084 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T05:01:28.084 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 5.1 M 2023-11-25T05:01:28.084 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:28.084 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T05:01:28.085 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:28.085 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T05:01:28.085 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:28.085 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 5.1 M 2023-11-25T05:01:28.085 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 24 M 2023-11-25T05:01:28.085 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T05:01:28.702 INFO:teuthology.orchestra.run.smithi007.stdout:libcephfs2-debuginfo-18.0.0-7506.gec928368.el9. 8.3 MB/s | 5.1 MB 00:00 2023-11-25T05:01:28.703 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:01:28.704 INFO:teuthology.orchestra.run.smithi007.stdout:Total 8.3 MB/s | 5.1 MB 00:00 2023-11-25T05:01:28.704 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T05:01:28.707 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T05:01:28.707 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T05:01:28.775 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T05:01:28.775 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T05:01:29.455 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T05:01:29.570 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-7506.gec928368.el9.x86 1/1 2023-11-25T05:01:30.158 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-7506.gec928368.el9.x86 1/1 2023-11-25T05:01:30.450 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-7506.gec928368.el9.x86 1/1 2023-11-25T05:01:30.450 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:30.450 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T05:01:30.450 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:01:30.450 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:30.450 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T05:01:30.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados2-debuginfo 2023-11-25T05:01:31.258 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:01 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T05:01:31.408 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T05:01:31.408 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:31.408 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T05:01:31.408 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:31.408 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T05:01:31.409 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 43 M 2023-11-25T05:01:31.409 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:31.409 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T05:01:31.409 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:31.409 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T05:01:31.409 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:31.409 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 43 M 2023-11-25T05:01:31.410 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 248 M 2023-11-25T05:01:31.410 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T05:01:34.605 INFO:teuthology.orchestra.run.smithi007.stdout:librados2-debuginfo-18.0.0-7506.gec928368.el9.x 14 MB/s | 43 MB 00:03 2023-11-25T05:01:34.606 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:01:34.606 INFO:teuthology.orchestra.run.smithi007.stdout:Total 14 MB/s | 43 MB 00:03 2023-11-25T05:01:34.606 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T05:01:34.611 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T05:01:34.611 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T05:01:34.924 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T05:01:34.925 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T05:01:37.270 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:01:37.559 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:01:37.560 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:37.560 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:01:37.560 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:01:37.560 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:37.560 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:01:37.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-fuse-debuginfo 2023-11-25T05:01:38.397 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:08:09 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T05:01:38.550 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T05:01:38.551 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:01:38.551 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T05:01:38.551 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:01:38.551 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T05:01:38.551 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 5.8 M 2023-11-25T05:01:38.551 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:38.551 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T05:01:38.552 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:01:38.552 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T05:01:38.552 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:38.552 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 5.8 M 2023-11-25T05:01:38.552 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 28 M 2023-11-25T05:01:38.552 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T05:01:39.222 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-fuse-debuginfo-18.0.0-7506.gec928368.el9.x 8.6 MB/s | 5.8 MB 00:00 2023-11-25T05:01:39.223 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:01:39.223 INFO:teuthology.orchestra.run.smithi090.stdout:Total 8.6 MB/s | 5.8 MB 00:00 2023-11-25T05:01:39.223 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:01:39.227 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:01:39.227 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:01:39.296 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:01:39.296 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:01:40.008 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:01:40.129 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T05:01:40.182 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T05:01:40.281 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados2-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T05:01:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T05:01:41.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T05:01:41.082 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:41.082 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:01:41.082 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:01:41.083 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:41.083 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:01:41.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados-devel-debuginfo 2023-11-25T05:01:41.872 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:08:12 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T05:01:42.015 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T05:01:42.016 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:01:42.016 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T05:01:42.016 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:01:42.016 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T05:01:42.017 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 108 k 2023-11-25T05:01:42.017 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:42.017 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T05:01:42.017 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:01:42.017 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T05:01:42.017 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:42.017 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 108 k 2023-11-25T05:01:42.017 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 440 k 2023-11-25T05:01:42.017 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T05:01:42.143 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T05:01:42.334 INFO:teuthology.orchestra.run.smithi090.stdout:librados-devel-debuginfo-18.0.0-7506.gec928368. 341 kB/s | 108 kB 00:00 2023-11-25T05:01:42.334 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:01:42.335 INFO:teuthology.orchestra.run.smithi090.stdout:Total 340 kB/s | 108 kB 00:00 2023-11-25T05:01:42.335 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:01:42.338 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:01:42.338 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:01:42.372 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:01:42.372 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:01:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T05:01:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:42.456 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T05:01:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:01:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:42.456 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T05:01:42.490 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:01:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-debuginfo-2:18.0.0-7506.gec928368.el9 1/1 2023-11-25T05:01:42.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librbd1-debuginfo 2023-11-25T05:01:43.005 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-7506.gec928368.el9 1/1 2023-11-25T05:01:43.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-7506.gec928368.el9 1/1 2023-11-25T05:01:43.270 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:43.270 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:01:43.270 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:01:43.270 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:43.270 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:01:43.277 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:13 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T05:01:43.430 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T05:01:43.431 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T05:01:43.431 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:43.431 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T05:01:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 50 M 2023-11-25T05:01:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:43.431 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T05:01:43.431 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:43.432 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T05:01:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:43.432 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 50 M 2023-11-25T05:01:43.432 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 294 M 2023-11-25T05:01:43.432 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T05:01:43.508 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs2-debuginfo 2023-11-25T05:01:44.104 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:08:15 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T05:01:44.249 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T05:01:44.250 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:01:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T05:01:44.250 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:01:44.250 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T05:01:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 5.1 M 2023-11-25T05:01:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:44.250 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T05:01:44.250 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:01:44.250 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T05:01:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:44.251 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 5.1 M 2023-11-25T05:01:44.251 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 24 M 2023-11-25T05:01:44.251 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T05:01:46.107 INFO:teuthology.orchestra.run.smithi007.stdout:librbd1-debuginfo-18.0.0-7506.gec928368.el9.x86 19 MB/s | 50 MB 00:02 2023-11-25T05:01:46.108 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:01:46.109 INFO:teuthology.orchestra.run.smithi007.stdout:Total 19 MB/s | 50 MB 00:02 2023-11-25T05:01:46.109 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T05:01:46.113 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T05:01:46.113 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T05:01:46.464 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T05:01:46.465 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T05:01:46.631 INFO:teuthology.orchestra.run.smithi090.stdout:libcephfs2-debuginfo-18.0.0-7506.gec928368.el9. 2.2 MB/s | 5.1 MB 00:02 2023-11-25T05:01:46.632 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:01:46.632 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.2 MB/s | 5.1 MB 00:02 2023-11-25T05:01:46.633 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:01:46.636 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:01:46.637 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:01:46.703 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:01:46.704 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:01:47.348 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:01:47.472 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-7506.gec928368.el9.x86 1/1 2023-11-25T05:01:48.141 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-7506.gec928368.el9.x86 1/1 2023-11-25T05:01:48.423 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-7506.gec928368.el9.x86 1/1 2023-11-25T05:01:48.423 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:48.424 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:01:48.424 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:01:48.424 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:48.424 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:01:48.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados2-debuginfo 2023-11-25T05:01:49.243 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:08:20 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T05:01:49.392 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T05:01:49.393 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:01:49.393 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T05:01:49.393 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:01:49.393 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T05:01:49.393 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 43 M 2023-11-25T05:01:49.393 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:49.393 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T05:01:49.393 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:01:49.394 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T05:01:49.394 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:01:49.394 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 43 M 2023-11-25T05:01:49.394 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 248 M 2023-11-25T05:01:49.394 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T05:01:52.135 INFO:teuthology.orchestra.run.smithi090.stdout:librados2-debuginfo-18.0.0-7506.gec928368.el9.x 16 MB/s | 43 MB 00:02 2023-11-25T05:01:52.135 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:01:52.136 INFO:teuthology.orchestra.run.smithi090.stdout:Total 16 MB/s | 43 MB 00:02 2023-11-25T05:01:52.136 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:01:52.140 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:01:52.140 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:01:52.440 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:01:52.440 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:01:52.555 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T05:01:52.659 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librbd1-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T05:01:54.782 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T05:01:55.079 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T05:01:55.080 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:55.080 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T05:01:55.080 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:01:55.080 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:55.080 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T05:01:55.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-cephfs-debuginfo 2023-11-25T05:01:55.877 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:25 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T05:01:56.024 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T05:01:56.025 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:56.025 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T05:01:56.025 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:56.026 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T05:01:56.026 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 615 k 2023-11-25T05:01:56.026 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:56.026 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T05:01:56.026 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:56.026 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T05:01:56.027 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:56.027 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 615 k 2023-11-25T05:01:56.027 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.1 M 2023-11-25T05:01:56.027 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T05:01:56.419 INFO:teuthology.orchestra.run.smithi007.stdout:python3-cephfs-debuginfo-18.0.0-7506.gec928368. 1.5 MB/s | 615 kB 00:00 2023-11-25T05:01:56.419 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:01:56.420 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.5 MB/s | 615 kB 00:00 2023-11-25T05:01:56.420 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T05:01:56.423 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T05:01:56.423 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T05:01:56.462 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T05:01:56.463 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T05:01:56.637 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T05:01:56.744 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-7506.gec928368.el9 1/1 2023-11-25T05:01:57.159 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-7506.gec928368.el9 1/1 2023-11-25T05:01:57.427 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:01:57.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-7506.gec928368.el9 1/1 2023-11-25T05:01:57.465 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:57.465 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T05:01:57.465 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:01:57.465 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:57.465 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T05:01:57.534 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados2-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T05:01:57.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rados-debuginfo 2023-11-25T05:01:58.284 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:28 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T05:01:58.434 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T05:01:58.435 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:58.435 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T05:01:58.435 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:58.435 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T05:01:58.435 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 771 k 2023-11-25T05:01:58.435 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:58.436 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T05:01:58.436 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:01:58.436 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T05:01:58.436 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:58.436 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 771 k 2023-11-25T05:01:58.436 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.7 M 2023-11-25T05:01:58.436 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T05:01:58.696 INFO:teuthology.orchestra.run.smithi007.stdout:python3-rados-debuginfo-18.0.0-7506.gec928368.e 2.9 MB/s | 771 kB 00:00 2023-11-25T05:01:58.697 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:01:58.697 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.9 MB/s | 771 kB 00:00 2023-11-25T05:01:58.697 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T05:01:58.701 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T05:01:58.701 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T05:01:58.742 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T05:01:58.742 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T05:01:58.945 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T05:01:59.057 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-debuginfo-2:18.0.0-7506.gec928368.el9. 1/1 2023-11-25T05:01:59.520 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-7506.gec928368.el9. 1/1 2023-11-25T05:01:59.804 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-7506.gec928368.el9. 1/1 2023-11-25T05:01:59.804 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:59.804 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T05:01:59.804 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:01:59.805 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:01:59.805 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T05:01:59.888 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T05:02:00.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rbd-debuginfo 2023-11-25T05:02:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-debuginfo-2:18.0.0-7506.gec928368.el9.x86_ 1/1 2023-11-25T05:02:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:00.165 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:02:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:02:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:00.166 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:02:00.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librbd1-debuginfo 2023-11-25T05:02:00.616 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:30 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T05:02:00.767 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T05:02:00.767 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:02:00.767 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T05:02:00.767 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:02:00.768 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T05:02:00.768 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 1.1 M 2023-11-25T05:02:00.768 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:00.768 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T05:02:00.768 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:02:00.768 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T05:02:00.768 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:00.768 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 1.1 M 2023-11-25T05:02:00.768 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 4.3 M 2023-11-25T05:02:00.768 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T05:02:00.977 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:08:31 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T05:02:01.091 INFO:teuthology.orchestra.run.smithi007.stdout:python3-rbd-debuginfo-18.0.0-7506.gec928368.el9 3.5 MB/s | 1.1 MB 00:00 2023-11-25T05:02:01.092 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:02:01.092 INFO:teuthology.orchestra.run.smithi007.stdout:Total 3.5 MB/s | 1.1 MB 00:00 2023-11-25T05:02:01.092 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T05:02:01.096 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T05:02:01.098 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T05:02:01.121 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T05:02:01.121 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T05:02:01.122 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:01.122 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T05:02:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 50 M 2023-11-25T05:02:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:01.122 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T05:02:01.122 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:01.122 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T05:02:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:01.123 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 50 M 2023-11-25T05:02:01.123 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 294 M 2023-11-25T05:02:01.123 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T05:02:01.139 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T05:02:01.139 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T05:02:01.368 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T05:02:01.460 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T05:02:01.883 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T05:02:02.180 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T05:02:02.181 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:02.181 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T05:02:02.181 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:02:02.181 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:02.181 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T05:02:02.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rgw-debuginfo 2023-11-25T05:02:03.007 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:33 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T05:02:03.158 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T05:02:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:02:03.159 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T05:02:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:02:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T05:02:03.159 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 337 k 2023-11-25T05:02:03.159 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T05:02:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:02:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T05:02:03.159 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 337 k 2023-11-25T05:02:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.0 M 2023-11-25T05:02:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T05:02:03.803 INFO:teuthology.orchestra.run.smithi007.stdout:python3-rgw-debuginfo-18.0.0-7506.gec928368.el9 524 kB/s | 337 kB 00:00 2023-11-25T05:02:03.804 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:02:03.804 INFO:teuthology.orchestra.run.smithi007.stdout:Total 523 kB/s | 337 kB 00:00 2023-11-25T05:02:03.804 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T05:02:03.808 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T05:02:03.808 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T05:02:03.849 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T05:02:03.850 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T05:02:03.994 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T05:02:04.104 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T05:02:04.503 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T05:02:04.616 INFO:teuthology.orchestra.run.smithi090.stdout:librbd1-debuginfo-18.0.0-7506.gec928368.el9.x86 14 MB/s | 50 MB 00:03 2023-11-25T05:02:04.617 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:02:04.617 INFO:teuthology.orchestra.run.smithi090.stdout:Total 14 MB/s | 50 MB 00:03 2023-11-25T05:02:04.617 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:02:04.621 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:02:04.621 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:02:04.810 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T05:02:04.810 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:04.810 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T05:02:04.811 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:02:04.811 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:04.811 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T05:02:04.959 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:02:04.959 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:02:05.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-fuse-debuginfo 2023-11-25T05:02:05.634 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:35 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T05:02:05.785 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T05:02:05.786 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:02:05.786 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T05:02:05.786 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:02:05.786 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T05:02:05.787 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 506 k 2023-11-25T05:02:05.787 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:05.787 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T05:02:05.787 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:02:05.787 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T05:02:05.787 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:05.787 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 506 k 2023-11-25T05:02:05.787 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.5 M 2023-11-25T05:02:05.787 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T05:02:06.131 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-fuse-debuginfo-18.0.0-7506.gec928368.el9.x8 1.4 MB/s | 506 kB 00:00 2023-11-25T05:02:06.132 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:02:06.132 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.4 MB/s | 506 kB 00:00 2023-11-25T05:02:06.132 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T05:02:06.136 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T05:02:06.136 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T05:02:06.175 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T05:02:06.176 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T05:02:06.339 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T05:02:06.449 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:02:06.894 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:02:07.187 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:02:07.188 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:07.188 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T05:02:07.190 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:02:07.190 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:07.193 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T05:02:07.389 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-mirror-debuginfo 2023-11-25T05:02:08.004 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:38 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T05:02:08.154 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T05:02:08.155 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:02:08.155 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T05:02:08.155 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:02:08.155 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T05:02:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 50 M 2023-11-25T05:02:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:08.156 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T05:02:08.156 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:02:08.156 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T05:02:08.156 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:08.156 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 50 M 2023-11-25T05:02:08.156 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 284 M 2023-11-25T05:02:08.156 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T05:02:10.584 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-mirror-debuginfo-18.0.0-7506.gec928368.el9. 20 MB/s | 50 MB 00:02 2023-11-25T05:02:10.585 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:02:10.585 INFO:teuthology.orchestra.run.smithi007.stdout:Total 20 MB/s | 50 MB 00:02 2023-11-25T05:02:10.586 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T05:02:10.590 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T05:02:10.590 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T05:02:10.711 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:02:10.807 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librbd1-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T05:02:10.932 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T05:02:10.932 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T05:02:13.422 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T05:02:13.706 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T05:02:13.706 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:13.706 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:02:13.706 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:02:13.706 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:13.706 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:02:13.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-cephfs-debuginfo 2023-11-25T05:02:14.540 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:08:45 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T05:02:14.685 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T05:02:14.686 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:14.686 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T05:02:14.686 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:14.687 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T05:02:14.687 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 615 k 2023-11-25T05:02:14.687 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:14.687 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T05:02:14.687 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:14.687 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T05:02:14.688 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:14.688 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 615 k 2023-11-25T05:02:14.688 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 2.1 M 2023-11-25T05:02:14.688 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T05:02:14.953 INFO:teuthology.orchestra.run.smithi090.stdout:python3-cephfs-debuginfo-18.0.0-7506.gec928368. 2.2 MB/s | 615 kB 00:00 2023-11-25T05:02:14.954 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:02:14.955 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.2 MB/s | 615 kB 00:00 2023-11-25T05:02:14.955 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:02:14.958 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:02:14.958 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:02:14.996 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:02:14.996 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:02:15.156 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:02:15.261 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-7506.gec928368.el9 1/1 2023-11-25T05:02:15.755 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-7506.gec928368.el9 1/1 2023-11-25T05:02:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-7506.gec928368.el9 1/1 2023-11-25T05:02:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:16.057 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:02:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:02:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:16.057 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:02:16.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rados-debuginfo 2023-11-25T05:02:16.877 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:08:47 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T05:02:16.943 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T05:02:17.018 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T05:02:17.019 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:17.019 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T05:02:17.019 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:17.019 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T05:02:17.019 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 771 k 2023-11-25T05:02:17.019 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:17.020 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T05:02:17.020 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:17.020 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T05:02:17.020 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:17.020 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 771 k 2023-11-25T05:02:17.020 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 2.7 M 2023-11-25T05:02:17.021 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T05:02:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-7506.gec928368.el9.x86 1/1 2023-11-25T05:02:17.458 INFO:teuthology.orchestra.run.smithi090.stdout:python3-rados-debuginfo-18.0.0-7506.gec928368.e 1.7 MB/s | 771 kB 00:00 2023-11-25T05:02:17.459 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:02:17.459 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.7 MB/s | 771 kB 00:00 2023-11-25T05:02:17.459 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:02:17.463 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:02:17.463 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:02:17.501 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:02:17.501 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:02:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:02:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-debuginfo-2:18.0.0-7506.gec928368.el9. 1/1 2023-11-25T05:02:18.261 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-7506.gec928368.el9. 1/1 2023-11-25T05:02:18.537 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-7506.gec928368.el9. 1/1 2023-11-25T05:02:18.537 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:18.537 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:02:18.537 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:02:18.538 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:18.538 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:02:18.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rbd-debuginfo 2023-11-25T05:02:19.111 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-7506.gec928368.el9.x86 1/1 2023-11-25T05:02:19.344 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:08:50 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T05:02:19.417 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-7506.gec928368.el9.x86 1/1 2023-11-25T05:02:19.418 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:19.418 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T05:02:19.418 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:02:19.418 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:19.418 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T05:02:19.490 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T05:02:19.490 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T05:02:19.491 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:19.491 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T05:02:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 1.1 M 2023-11-25T05:02:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:19.491 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T05:02:19.492 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:19.492 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T05:02:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:19.492 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 1.1 M 2023-11-25T05:02:19.492 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 4.3 M 2023-11-25T05:02:19.492 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T05:02:19.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-nbd-debuginfo 2023-11-25T05:02:19.946 INFO:teuthology.orchestra.run.smithi090.stdout:python3-rbd-debuginfo-18.0.0-7506.gec928368.el9 2.5 MB/s | 1.1 MB 00:00 2023-11-25T05:02:19.946 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:02:19.947 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.5 MB/s | 1.1 MB 00:00 2023-11-25T05:02:19.947 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:02:19.950 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:02:19.950 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:02:19.990 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:02:19.991 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:02:20.210 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:02:20.229 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:50 ago on Sat 25 Nov 2023 04:53:30 AM UTC. 2023-11-25T05:02:20.355 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T05:02:20.382 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-25T05:02:20.382 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:02:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-25T05:02:20.382 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:02:20.382 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-25T05:02:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 1.2 M 2023-11-25T05:02:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:20.382 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-25T05:02:20.383 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-25T05:02:20.383 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-25T05:02:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:20.383 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 1.2 M 2023-11-25T05:02:20.383 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 5.4 M 2023-11-25T05:02:20.383 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-25T05:02:20.828 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-nbd-debuginfo-18.0.0-7506.gec928368.el9.x86 2.8 MB/s | 1.2 MB 00:00 2023-11-25T05:02:20.828 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:02:20.829 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.8 MB/s | 1.2 MB 00:00 2023-11-25T05:02:20.829 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-25T05:02:20.830 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T05:02:20.833 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-25T05:02:20.833 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-25T05:02:20.876 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-25T05:02:20.876 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-25T05:02:21.111 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T05:02:21.112 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:21.112 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:02:21.112 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:02:21.112 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:21.112 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:02:21.132 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-25T05:02:21.230 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T05:02:21.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rgw-debuginfo 2023-11-25T05:02:21.671 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T05:02:21.935 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:08:52 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T05:02:21.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T05:02:21.966 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:21.966 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-25T05:02:21.966 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:02:21.966 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:21.967 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-25T05:02:22.079 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T05:02:22.080 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T05:02:22.080 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:22.080 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T05:02:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 337 k 2023-11-25T05:02:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:22.081 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T05:02:22.081 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:22.081 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T05:02:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:22.081 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 337 k 2023-11-25T05:02:22.081 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 1.0 M 2023-11-25T05:02:22.081 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T05:02:22.180 DEBUG:teuthology.parallel:result is None 2023-11-25T05:02:22.485 INFO:teuthology.orchestra.run.smithi090.stdout:python3-rgw-debuginfo-18.0.0-7506.gec928368.el9 836 kB/s | 337 kB 00:00 2023-11-25T05:02:22.485 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:02:22.485 INFO:teuthology.orchestra.run.smithi090.stdout:Total 833 kB/s | 337 kB 00:00 2023-11-25T05:02:22.486 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:02:22.489 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:02:22.489 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:02:22.524 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:02:22.524 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:02:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:02:22.776 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T05:02:23.215 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T05:02:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-7506.gec928368.el9.x8 1/1 2023-11-25T05:02:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:23.493 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:02:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:02:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:23.494 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:02:23.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-fuse-debuginfo 2023-11-25T05:02:24.296 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:08:55 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T05:02:24.441 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T05:02:24.441 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T05:02:24.441 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:24.441 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T05:02:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 506 k 2023-11-25T05:02:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:24.442 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T05:02:24.442 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:24.442 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T05:02:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:24.442 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 506 k 2023-11-25T05:02:24.442 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 2.5 M 2023-11-25T05:02:24.443 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T05:02:24.855 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-fuse-debuginfo-18.0.0-7506.gec928368.el9.x8 1.2 MB/s | 506 kB 00:00 2023-11-25T05:02:24.856 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:02:24.856 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.2 MB/s | 506 kB 00:00 2023-11-25T05:02:24.857 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:02:24.860 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:02:24.860 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:02:24.897 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:02:24.897 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:02:25.052 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:02:25.147 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:02:25.630 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:02:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_6 1/1 2023-11-25T05:02:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:25.910 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:02:25.910 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:02:25.910 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:25.910 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:02:26.129 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-mirror-debuginfo 2023-11-25T05:02:26.707 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:08:57 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T05:02:26.852 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T05:02:26.852 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:26.852 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T05:02:26.853 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:26.853 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T05:02:26.853 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 50 M 2023-11-25T05:02:26.853 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:26.853 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T05:02:26.853 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:26.853 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T05:02:26.853 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:26.853 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 50 M 2023-11-25T05:02:26.853 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 284 M 2023-11-25T05:02:26.853 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T05:02:31.452 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-mirror-debuginfo-18.0.0-7506.gec928368.el9. 11 MB/s | 50 MB 00:04 2023-11-25T05:02:31.453 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:02:31.454 INFO:teuthology.orchestra.run.smithi090.stdout:Total 11 MB/s | 50 MB 00:04 2023-11-25T05:02:31.454 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:02:31.458 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:02:31.458 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:02:31.785 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:02:31.785 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:02:37.485 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:02:37.593 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-7506.gec928368.el9.x86 1/1 2023-11-25T05:02:40.110 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-7506.gec928368.el9.x86 1/1 2023-11-25T05:02:40.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-7506.gec928368.el9.x86 1/1 2023-11-25T05:02:40.403 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:40.403 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:02:40.403 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:02:40.403 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:40.403 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:02:40.664 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-nbd-debuginfo 2023-11-25T05:02:41.242 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:09:12 ago on Sat 25 Nov 2023 04:53:29 AM UTC. 2023-11-25T05:02:41.385 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-11-25T05:02:41.386 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:41.386 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2023-11-25T05:02:41.386 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:41.387 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-11-25T05:02:41.387 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-7506.gec928368.el9 ceph 1.2 M 2023-11-25T05:02:41.387 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:41.387 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-11-25T05:02:41.387 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2023-11-25T05:02:41.387 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2023-11-25T05:02:41.387 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:41.387 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 1.2 M 2023-11-25T05:02:41.387 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 5.4 M 2023-11-25T05:02:41.387 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-11-25T05:02:41.831 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-nbd-debuginfo-18.0.0-7506.gec928368.el9.x86 2.8 MB/s | 1.2 MB 00:00 2023-11-25T05:02:41.831 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-11-25T05:02:41.832 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.8 MB/s | 1.2 MB 00:00 2023-11-25T05:02:41.832 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-11-25T05:02:41.835 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-11-25T05:02:41.835 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-11-25T05:02:41.876 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-11-25T05:02:41.876 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-11-25T05:02:42.111 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-11-25T05:02:42.214 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T05:02:42.702 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T05:02:42.994 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 1/1 2023-11-25T05:02:42.994 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:42.995 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-11-25T05:02:42.995 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-debuginfo-2:18.0.0-7506.gec928368.el9.x86_64 2023-11-25T05:02:42.995 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:42.995 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-11-25T05:02:43.246 DEBUG:teuthology.parallel:result is None 2023-11-25T05:02:43.247 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ec928368edcc5fc71884a56fd9fd821b148f4e30 2023-11-25T05:02:43.396 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-25T05:02:43.422 INFO:teuthology.orchestra.run.smithi007.stdout:18.0.0-7506.gec928368.el9 2023-11-25T05:02:43.422 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7506.gec928368.el9 2023-11-25T05:02:43.423 INFO:teuthology.task.install:The correct ceph version 18.0.0-7506.gec928368 is installed. 2023-11-25T05:02:43.423 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ec928368edcc5fc71884a56fd9fd821b148f4e30 2023-11-25T05:02:43.643 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-25T05:02:43.664 INFO:teuthology.orchestra.run.smithi090.stdout:18.0.0-7506.gec928368.el9 2023-11-25T05:02:43.664 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7506.gec928368.el9 2023-11-25T05:02:43.665 INFO:teuthology.task.install:The correct ceph version 18.0.0-7506.gec928368 is installed. 2023-11-25T05:02:43.665 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-25T05:02:43.665 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:02:43.665 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-25T05:02:43.702 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:02:43.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-25T05:02:43.731 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-25T05:02:43.732 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:02:43.732 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2023-11-25T05:02:43.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-25T05:02:43.851 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:02:43.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2023-11-25T05:02:43.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-25T05:02:43.946 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-25T05:02:43.946 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:02:43.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-25T05:02:43.979 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-25T05:02:44.052 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:02:44.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-25T05:02:44.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-25T05:02:44.146 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-25T05:02:44.146 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:02:44.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2023-11-25T05:02:44.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-25T05:02:44.249 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:02:44.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2023-11-25T05:02:44.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-25T05:02:44.343 INFO:teuthology.run_tasks:Running task ceph... 2023-11-25T05:02:44.409 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-25T05:02:44.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /var/log/ceph 2023-11-25T05:02:44.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /var/log/ceph 2023-11-25T05:02:44.441 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-25T05:02:44.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-25T05:02:44.485 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-25T05:02:44.514 INFO:tasks.ceph:Creating extra log directories... 2023-11-25T05:02:44.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-25T05:02:44.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-25T05:02:44.596 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-25T05:02:44.597 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-25T05:02:44.597 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196', 'branch': 'main', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7466196', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'ec928368edcc5fc71884a56fd9fd821b148f4e30', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'ec928368edcc5fc71884a56fd9fd821b148f4e30'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': 'ec928368edcc5fc71884a56fd9fd821b148f4e30'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 77, 'sha1': 'ec928368edcc5fc71884a56fd9fd821b148f4e30', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_ec928368edcc5fc71884a56fd9fd821b148f4e30/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'ec928368edcc5fc71884a56fd9fd821b148f4e30', 'targets': {'smithi007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIQLzUj5DoDXok6ckR54zrIrE0YNr5/E8OHdWoHANIrAr9UBNH2fLFLMlj9n4BT4u5otCq4Gq1yMp/4oCqYCggY=', 'smithi090.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFtIGrbVuXwbnaMq4SVU04AhZvexOMXUoe+9Sjmpom43axHDTqNK2H6t0ELoI7LvcVNKRAUXDUS45FAfH/4JmEE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '2442239d2653456406c25ae0c71b689c8f2657b6', 'timestamp': '2023-11-24_21:18:30', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-11-25T05:02:44.597 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-25T05:02:44.629 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-25T05:02:44.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/run/ceph 2023-11-25T05:02:44.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/run/ceph 2023-11-25T05:02:44.713 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:02:44.714 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2023-11-25T05:02:44.769 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-25T05:02:44.769 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2023-11-25T05:02:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-25T05:02:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-25T05:02:44.825 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-11-25T05:02:44.825 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-25T05:02:44.825 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-11-25T05:02:44.825 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-11-25 04:55:14.853275040 +0000 2023-11-25T05:02:44.826 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-11-25 04:52:40.241930517 +0000 2023-11-25T05:02:44.826 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-11-25 04:52:40.241930517 +0000 2023-11-25T05:02:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-11-25 04:52:40.241930517 +0000 2023-11-25T05:02:44.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-25T05:02:44.897 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-11-25T05:02:44.897 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-11-25T05:02:44.897 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000105077 s, 4.9 MB/s 2023-11-25T05:02:44.897 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-25T05:02:44.953 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2023-11-25T05:02:45.008 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-25T05:02:45.008 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-25T05:02:45.009 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-11-25T05:02:45.009 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-25T05:02:45.009 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-11-25T05:02:45.009 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-11-25 04:55:14.854275025 +0000 2023-11-25T05:02:45.009 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-11-25 04:52:40.748922810 +0000 2023-11-25T05:02:45.009 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-11-25 04:52:40.748922810 +0000 2023-11-25T05:02:45.009 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-11-25 04:52:40.748922810 +0000 2023-11-25T05:02:45.009 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-25T05:02:45.077 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-11-25T05:02:45.077 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-11-25T05:02:45.077 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 8.3512e-05 s, 6.1 MB/s 2023-11-25T05:02:45.078 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-25T05:02:45.136 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2023-11-25T05:02:45.192 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-25T05:02:45.192 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-25T05:02:45.192 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-11-25T05:02:45.192 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-25T05:02:45.192 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-11-25T05:02:45.192 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-11-25 04:55:14.854275025 +0000 2023-11-25T05:02:45.192 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-11-25 04:52:41.279914738 +0000 2023-11-25T05:02:45.192 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-11-25 04:52:41.279914738 +0000 2023-11-25T05:02:45.193 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-11-25 04:52:41.279914738 +0000 2023-11-25T05:02:45.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-25T05:02:45.261 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-11-25T05:02:45.261 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-11-25T05:02:45.261 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 7.8222e-05 s, 6.5 MB/s 2023-11-25T05:02:45.262 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-25T05:02:45.322 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2023-11-25T05:02:45.377 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-25T05:02:45.377 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-25T05:02:45.377 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-11-25T05:02:45.377 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-25T05:02:45.378 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-11-25T05:02:45.378 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-11-25 04:55:14.854275025 +0000 2023-11-25T05:02:45.378 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-11-25 04:52:41.779907138 +0000 2023-11-25T05:02:45.378 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-11-25 04:52:41.779907138 +0000 2023-11-25T05:02:45.378 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-11-25 04:52:41.779907138 +0000 2023-11-25T05:02:45.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-25T05:02:45.446 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-11-25T05:02:45.446 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-11-25T05:02:45.447 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.8015e-05 s, 5.2 MB/s 2023-11-25T05:02:45.448 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-25T05:02:45.504 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-25T05:02:45.504 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:02:45.504 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2023-11-25T05:02:45.520 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-25T05:02:45.520 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2023-11-25T05:02:45.576 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-25T05:02:45.576 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-25T05:02:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-11-25T05:02:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-25T05:02:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2023-11-25T05:02:45.577 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-11-25 04:55:18.250367347 +0000 2023-11-25T05:02:45.577 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-11-25 04:52:33.285202725 +0000 2023-11-25T05:02:45.577 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-11-25 04:52:33.285202725 +0000 2023-11-25T05:02:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2023-11-25 04:52:33.285202725 +0000 2023-11-25T05:02:45.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-25T05:02:45.643 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-11-25T05:02:45.643 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-11-25T05:02:45.643 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000210113 s, 2.4 MB/s 2023-11-25T05:02:45.644 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-25T05:02:45.701 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2023-11-25T05:02:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-25T05:02:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-25T05:02:45.757 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-11-25T05:02:45.757 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-25T05:02:45.757 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2023-11-25T05:02:45.757 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-11-25 04:55:18.250367347 +0000 2023-11-25T05:02:45.757 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-11-25 04:52:33.782194808 +0000 2023-11-25T05:02:45.757 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-11-25 04:52:33.782194808 +0000 2023-11-25T05:02:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2023-11-25 04:52:33.782194808 +0000 2023-11-25T05:02:45.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-25T05:02:45.823 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-11-25T05:02:45.824 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-11-25T05:02:45.824 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000212137 s, 2.4 MB/s 2023-11-25T05:02:45.825 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-25T05:02:45.881 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2023-11-25T05:02:45.936 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-25T05:02:45.937 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-25T05:02:45.937 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-11-25T05:02:45.937 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-25T05:02:45.937 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2023-11-25T05:02:45.937 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-11-25 04:55:18.250367347 +0000 2023-11-25T05:02:45.937 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-11-25 04:52:34.275186955 +0000 2023-11-25T05:02:45.937 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-11-25 04:52:34.275186955 +0000 2023-11-25T05:02:45.937 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2023-11-25 04:52:34.275186955 +0000 2023-11-25T05:02:45.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-25T05:02:46.002 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-11-25T05:02:46.002 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-11-25T05:02:46.002 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000209847 s, 2.4 MB/s 2023-11-25T05:02:46.003 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-25T05:02:46.059 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2023-11-25T05:02:46.114 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-25T05:02:46.114 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-25T05:02:46.114 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-11-25T05:02:46.114 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-25T05:02:46.114 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2023-11-25T05:02:46.114 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-11-25 04:55:18.251367331 +0000 2023-11-25T05:02:46.114 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-11-25 04:52:34.768179102 +0000 2023-11-25T05:02:46.114 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-11-25 04:52:34.768179102 +0000 2023-11-25T05:02:46.115 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2023-11-25 04:52:34.768179102 +0000 2023-11-25T05:02:46.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-25T05:02:46.179 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-11-25T05:02:46.180 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-11-25T05:02:46.180 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000187803 s, 2.7 MB/s 2023-11-25T05:02:46.181 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-25T05:02:46.237 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-25T05:02:46.237 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-25T05:02:46.237 INFO:tasks.ceph:Generating config... 2023-11-25T05:02:46.238 INFO:tasks.ceph:[client] debug monc = 20 2023-11-25T05:02:46.238 INFO:tasks.ceph:[client] debug ms = 1 2023-11-25T05:02:46.238 INFO:tasks.ceph:[client] debug objecter = 20 2023-11-25T05:02:46.238 INFO:tasks.ceph:[client] debug rados = 20 2023-11-25T05:02:46.239 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-11-25T05:02:46.239 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-11-25T05:02:46.239 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-11-25T05:02:46.239 INFO:tasks.ceph:[global] ms type = async 2023-11-25T05:02:46.239 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-11-25T05:02:46.239 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-25T05:02:46.239 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-25T05:02:46.239 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-25T05:02:46.239 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-25T05:02:46.239 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-25T05:02:46.239 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-11-25T05:02:46.239 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-11-25T05:02:46.240 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-11-25T05:02:46.240 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-11-25T05:02:46.240 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-11-25T05:02:46.240 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-11-25T05:02:46.240 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-11-25T05:02:46.240 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-11-25T05:02:46.240 INFO:tasks.ceph:[osd] bdev async discard = True 2023-11-25T05:02:46.240 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-11-25T05:02:46.240 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-11-25T05:02:46.240 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-25T05:02:46.240 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-25T05:02:46.240 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-11-25T05:02:46.240 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-11-25T05:02:46.241 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-11-25T05:02:46.241 INFO:tasks.ceph:[osd] debug monc = 20 2023-11-25T05:02:46.241 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-25T05:02:46.241 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-25T05:02:46.241 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-25T05:02:46.241 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-25T05:02:46.241 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-25T05:02:46.241 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-25T05:02:46.241 INFO:tasks.ceph:[osd] osd class default list = * 2023-11-25T05:02:46.241 INFO:tasks.ceph:[osd] osd class load list = * 2023-11-25T05:02:46.241 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2023-11-25T05:02:46.241 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-11-25T05:02:46.242 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-11-25T05:02:46.242 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-25T05:02:46.242 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-11-25T05:02:46.242 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-11-25T05:02:46.242 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-11-25T05:02:46.242 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-25T05:02:46.242 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-11-25T05:02:46.242 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-11-25T05:02:46.242 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-11-25T05:02:46.243 INFO:tasks.ceph:Setting up mon.a... 2023-11-25T05:02:46.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-25T05:02:46.313 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.keyring 2023-11-25T05:02:46.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-25T05:02:46.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-25T05:02:46.396 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.7'), ('mon.c', '[v2:172.21.15.7:3301,v1:172.21.15.7:6790]'), ('mon.b', '172.21.15.90')] 2023-11-25T05:02:46.397 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.7,[v2:172.21.15.7:3301,v1:172.21.15.7:6790],172.21.15.90', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-11-25T05:02:46.398 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:02:46.398 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-25T05:02:46.451 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.7 --addv c '[v2:172.21.15.7:3301,v1:172.21.15.7:6790]' --add b 172.21.15.90 --print /home/ubuntu/cephtest/ceph.monmap 2023-11-25T05:02:46.536 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-25T05:02:46.537 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: generated fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:02:46.537 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = pacific 2023-11-25T05:02:46.537 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 0 2023-11-25T05:02:46.537 INFO:teuthology.orchestra.run.smithi007.stdout:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:02:46.537 INFO:teuthology.orchestra.run.smithi007.stdout:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:02:46.537 INFO:teuthology.orchestra.run.smithi007.stdout:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:02:46.537 INFO:teuthology.orchestra.run.smithi007.stdout:min_mon_release 16 (pacific) 2023-11-25T05:02:46.537 INFO:teuthology.orchestra.run.smithi007.stdout:election_strategy: 1 2023-11-25T05:02:46.537 INFO:teuthology.orchestra.run.smithi007.stdout:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:02:46.537 INFO:teuthology.orchestra.run.smithi007.stdout:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:02:46.537 INFO:teuthology.orchestra.run.smithi007.stdout:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:02:46.538 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-11-25T05:02:46.539 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-25T05:02:46.554 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 59cf1eff-816d-4eda-b513-a2de7ea218f7... 2023-11-25T05:02:46.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-25T05:02:46.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-25T05:02:46.648 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2023-11-25T05:02:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: chdir = "" 2023-11-25T05:02:46.648 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-25T05:02:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: auth supported = cephx 2023-11-25T05:02:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: filestore xattr use omap = true 2023-11-25T05:02:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: mon clock drift allowed = 1.000 2023-11-25T05:02:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: osd crush chooseleaf type = 0 2023-11-25T05:02:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: auth debug = true 2023-11-25T05:02:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on old message = true 2023-11-25T05:02:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on bug = true 2023-11-25T05:02:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.649 INFO:teuthology.orchestra.run.smithi090.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-25T05:02:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: mon pg warn max object skew = 0 2023-11-25T05:02:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: # disable pg_autoscaler by default for new pools 2023-11-25T05:02:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-25T05:02:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default size = 2 2023-11-25T05:02:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow primary affinity = true 2023-11-25T05:02:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow pg remap = true 2023-11-25T05:02:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on legacy crush tunables = false 2023-11-25T05:02:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on crush straw calc version zero = false 2023-11-25T05:02:46.650 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on no sortbitwise = false 2023-11-25T05:02:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on osd down out interval zero = false 2023-11-25T05:02:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on too few osds = false 2023-11-25T05:02:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-25T05:02:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-25T05:02:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: mon_allow_pool_size_one = true 2023-11-25T05:02:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-25T05:02:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: osd default data pool replay window = 5 2023-11-25T05:02:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: mon allow pool delete = true 2023-11-25T05:02:46.651 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: mon cluster log file level = debug 2023-11-25T05:02:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: debug asserts on shutdown = true 2023-11-25T05:02:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: mon health detail to clog = false 2023-11-25T05:02:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: mon host = "172.21.15.7,[v2:172.21.15.7:3301,v1:172.21.15.7:6790],172.21.15.90" 2023-11-25T05:02:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: mon client directed command retry = 5 2023-11-25T05:02:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: mon election default strategy = 1 2023-11-25T05:02:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: ms inject socket failures = 5000 2023-11-25T05:02:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: ms type = async 2023-11-25T05:02:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: osd heartbeat grace = 80 2023-11-25T05:02:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:02:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.652 INFO:teuthology.orchestra.run.smithi090.stdout:[osd] 2023-11-25T05:02:46.652 INFO:teuthology.orchestra.run.smithi090.stdout: osd journal size = 100 2023-11-25T05:02:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub load threshold = 5.0 2023-11-25T05:02:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub max interval = 600 2023-11-25T05:02:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: osd mclock profile = high_recovery_ops 2023-11-25T05:02:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: osd recover clone overlap = true 2023-11-25T05:02:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: osd recovery max chunk = 1048576 2023-11-25T05:02:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug shutdown = true 2023-11-25T05:02:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug op order = true 2023-11-25T05:02:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify stray on activate = true 2023-11-25T05:02:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug trim objects = true 2023-11-25T05:02:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: osd open classes on start = true 2023-11-25T05:02:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug pg log writeout = true 2023-11-25T05:02:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: osd deep scrub update digest min age = 30 2023-11-25T05:02:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: osd map max advance = 10 2023-11-25T05:02:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: journal zero on create = true 2023-11-25T05:02:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: filestore ondisk finisher threads = 3 2023-11-25T05:02:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: filestore apply finisher threads = 3 2023-11-25T05:02:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: bdev debug aio = true 2023-11-25T05:02:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug misdirected ops = true 2023-11-25T05:02:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: bdev async discard = True 2023-11-25T05:02:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: bdev enable discard = True 2023-11-25T05:02:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore allocator = bitmap 2023-11-25T05:02:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore block size = 96636764160 2023-11-25T05:02:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore fsck on mount = True 2023-11-25T05:02:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore zero block detection = True 2023-11-25T05:02:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluefs = 1 2023-11-25T05:02:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluestore = 1 2023-11-25T05:02:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: debug monc = 20 2023-11-25T05:02:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2023-11-25T05:02:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: debug osd = 20 2023-11-25T05:02:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: debug rocksdb = 10 2023-11-25T05:02:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd backfillfull_ratio = 0.85 2023-11-25T05:02:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd full ratio = 0.9 2023-11-25T05:02:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd nearfull ratio = 0.8 2023-11-25T05:02:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: osd class default list = * 2023-11-25T05:02:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: osd class load list = * 2023-11-25T05:02:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: osd client watch timeout = 120 2023-11-25T05:02:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify cached snaps = True 2023-11-25T05:02:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify missing on start = True 2023-11-25T05:02:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: osd failsafe full ratio = 0.95 2023-11-25T05:02:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: osd fast shutdown = False 2023-11-25T05:02:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: osd mclock override recovery settings = True 2023-11-25T05:02:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: osd objectstore = bluestore 2023-11-25T05:02:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: osd op queue = debug_random 2023-11-25T05:02:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: osd op queue cut off = debug_random 2023-11-25T05:02:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: osd shutdown pgref assert = True 2023-11-25T05:02:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.657 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr] 2023-11-25T05:02:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2023-11-25T05:02:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: debug mgr = 20 2023-11-25T05:02:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2023-11-25T05:02:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2023-11-25T05:02:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2023-11-25T05:02:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2023-11-25T05:02:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: mgr/telemetry/nag = false 2023-11-25T05:02:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.658 INFO:teuthology.orchestra.run.smithi090.stdout:[mon] 2023-11-25T05:02:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2023-11-25T05:02:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2023-11-25T05:02:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: debug paxos = 20 2023-11-25T05:02:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2023-11-25T05:02:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: mon data avail warn = 5 2023-11-25T05:02:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: mon mgr mkfs grace = 240 2023-11-25T05:02:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2023-11-25T05:02:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd reporter subtree level = osd 2023-11-25T05:02:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd prime pg temp = true 2023-11-25T05:02:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2023-11-25T05:02:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-25T05:02:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: auth mon ticket ttl = 660 # 11m 2023-11-25T05:02:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: auth service ticket ttl = 240 # 4m 2023-11-25T05:02:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: # don't complain about insecure global_id in the test suite 2023-11-25T05:02:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-25T05:02:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-25T05:02:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: # 1m isn't quite enough 2023-11-25T05:02:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: mon_down_mkfs_grace = 2m 2023-11-25T05:02:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_filestore_osds = false 2023-11-25T05:02:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: mon min osdmap epochs = 50 2023-11-25T05:02:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd crush smoke test = False 2023-11-25T05:02:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: mon osdmap full prune interval = 2 2023-11-25T05:02:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: mon osdmap full prune min = 15 2023-11-25T05:02:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: mon osdmap full prune txsize = 2 2023-11-25T05:02:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: mon scrub interval = 300 2023-11-25T05:02:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on pool no app = False 2023-11-25T05:02:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: paxos service trim min = 10 2023-11-25T05:02:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-25T05:02:46.660 INFO:teuthology.orchestra.run.smithi090.stdout:[client] 2023-11-25T05:02:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: rgw cache enabled = true 2023-11-25T05:02:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable ops log = true 2023-11-25T05:02:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable usage log = true 2023-11-25T05:02:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-25T05:02:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-25T05:02:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: debug monc = 20 2023-11-25T05:02:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2023-11-25T05:02:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: debug objecter = 20 2023-11-25T05:02:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: debug rados = 20 2023-11-25T05:02:46.661 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.a] 2023-11-25T05:02:46.661 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.c] 2023-11-25T05:02:46.661 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.b] 2023-11-25T05:02:46.662 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2023-11-25T05:02:46.662 INFO:teuthology.orchestra.run.smithi007.stdout: chdir = "" 2023-11-25T05:02:46.662 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-25T05:02:46.662 INFO:teuthology.orchestra.run.smithi007.stdout: auth supported = cephx 2023-11-25T05:02:46.662 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.662 INFO:teuthology.orchestra.run.smithi007.stdout: filestore xattr use omap = true 2023-11-25T05:02:46.662 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.662 INFO:teuthology.orchestra.run.smithi007.stdout: mon clock drift allowed = 1.000 2023-11-25T05:02:46.663 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.663 INFO:teuthology.orchestra.run.smithi007.stdout: osd crush chooseleaf type = 0 2023-11-25T05:02:46.663 INFO:teuthology.orchestra.run.smithi007.stdout: auth debug = true 2023-11-25T05:02:46.663 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.663 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on old message = true 2023-11-25T05:02:46.663 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on bug = true 2023-11-25T05:02:46.663 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.663 INFO:teuthology.orchestra.run.smithi007.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-25T05:02:46.663 INFO:teuthology.orchestra.run.smithi007.stdout: mon pg warn max object skew = 0 2023-11-25T05:02:46.663 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.663 INFO:teuthology.orchestra.run.smithi007.stdout: # disable pg_autoscaler by default for new pools 2023-11-25T05:02:46.663 INFO:teuthology.orchestra.run.smithi007.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-25T05:02:46.664 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.664 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default size = 2 2023-11-25T05:02:46.664 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.664 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow primary affinity = true 2023-11-25T05:02:46.664 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow pg remap = true 2023-11-25T05:02:46.664 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on legacy crush tunables = false 2023-11-25T05:02:46.664 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on crush straw calc version zero = false 2023-11-25T05:02:46.664 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on no sortbitwise = false 2023-11-25T05:02:46.664 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on osd down out interval zero = false 2023-11-25T05:02:46.664 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on too few osds = false 2023-11-25T05:02:46.664 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-25T05:02:46.664 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-25T05:02:46.664 INFO:teuthology.orchestra.run.smithi007.stdout: mon_allow_pool_size_one = true 2023-11-25T05:02:46.665 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.665 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-25T05:02:46.665 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.665 INFO:teuthology.orchestra.run.smithi007.stdout: osd default data pool replay window = 5 2023-11-25T05:02:46.665 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.665 INFO:teuthology.orchestra.run.smithi007.stdout: mon allow pool delete = true 2023-11-25T05:02:46.665 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.665 INFO:teuthology.orchestra.run.smithi007.stdout: mon cluster log file level = debug 2023-11-25T05:02:46.665 INFO:teuthology.orchestra.run.smithi007.stdout: debug asserts on shutdown = true 2023-11-25T05:02:46.665 INFO:teuthology.orchestra.run.smithi007.stdout: mon health detail to clog = false 2023-11-25T05:02:46.665 INFO:teuthology.orchestra.run.smithi007.stdout: mon host = "172.21.15.7,[v2:172.21.15.7:3301,v1:172.21.15.7:6790],172.21.15.90" 2023-11-25T05:02:46.665 INFO:teuthology.orchestra.run.smithi007.stdout: mon client directed command retry = 5 2023-11-25T05:02:46.665 INFO:teuthology.orchestra.run.smithi007.stdout: mon election default strategy = 1 2023-11-25T05:02:46.666 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject socket failures = 5000 2023-11-25T05:02:46.666 INFO:teuthology.orchestra.run.smithi007.stdout: ms type = async 2023-11-25T05:02:46.666 INFO:teuthology.orchestra.run.smithi007.stdout: osd heartbeat grace = 80 2023-11-25T05:02:46.666 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:02:46.666 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.666 INFO:teuthology.orchestra.run.smithi007.stdout:[osd] 2023-11-25T05:02:46.666 INFO:teuthology.orchestra.run.smithi007.stdout: osd journal size = 100 2023-11-25T05:02:46.666 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.666 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub load threshold = 5.0 2023-11-25T05:02:46.666 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub max interval = 600 2023-11-25T05:02:46.666 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock profile = high_recovery_ops 2023-11-25T05:02:46.666 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.666 INFO:teuthology.orchestra.run.smithi007.stdout: osd recover clone overlap = true 2023-11-25T05:02:46.667 INFO:teuthology.orchestra.run.smithi007.stdout: osd recovery max chunk = 1048576 2023-11-25T05:02:46.667 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.667 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug shutdown = true 2023-11-25T05:02:46.667 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug op order = true 2023-11-25T05:02:46.667 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify stray on activate = true 2023-11-25T05:02:46.667 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug trim objects = true 2023-11-25T05:02:46.667 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.667 INFO:teuthology.orchestra.run.smithi007.stdout: osd open classes on start = true 2023-11-25T05:02:46.667 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug pg log writeout = true 2023-11-25T05:02:46.667 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.667 INFO:teuthology.orchestra.run.smithi007.stdout: osd deep scrub update digest min age = 30 2023-11-25T05:02:46.667 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.667 INFO:teuthology.orchestra.run.smithi007.stdout: osd map max advance = 10 2023-11-25T05:02:46.668 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.668 INFO:teuthology.orchestra.run.smithi007.stdout: journal zero on create = true 2023-11-25T05:02:46.668 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.668 INFO:teuthology.orchestra.run.smithi007.stdout: filestore ondisk finisher threads = 3 2023-11-25T05:02:46.668 INFO:teuthology.orchestra.run.smithi007.stdout: filestore apply finisher threads = 3 2023-11-25T05:02:46.668 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.668 INFO:teuthology.orchestra.run.smithi007.stdout: bdev debug aio = true 2023-11-25T05:02:46.668 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug misdirected ops = true 2023-11-25T05:02:46.668 INFO:teuthology.orchestra.run.smithi007.stdout: bdev async discard = True 2023-11-25T05:02:46.668 INFO:teuthology.orchestra.run.smithi007.stdout: bdev enable discard = True 2023-11-25T05:02:46.668 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore allocator = bitmap 2023-11-25T05:02:46.669 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore block size = 96636764160 2023-11-25T05:02:46.669 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore fsck on mount = True 2023-11-25T05:02:46.669 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore zero block detection = True 2023-11-25T05:02:46.669 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluefs = 1 2023-11-25T05:02:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluestore = 1 2023-11-25T05:02:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: debug monc = 20 2023-11-25T05:02:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-11-25T05:02:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 20 2023-11-25T05:02:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: debug rocksdb = 10 2023-11-25T05:02:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd backfillfull_ratio = 0.85 2023-11-25T05:02:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd full ratio = 0.9 2023-11-25T05:02:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd nearfull ratio = 0.8 2023-11-25T05:02:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: osd class default list = * 2023-11-25T05:02:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: osd class load list = * 2023-11-25T05:02:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: osd client watch timeout = 120 2023-11-25T05:02:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify cached snaps = True 2023-11-25T05:02:46.670 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify missing on start = True 2023-11-25T05:02:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: osd failsafe full ratio = 0.95 2023-11-25T05:02:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: osd fast shutdown = False 2023-11-25T05:02:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock override recovery settings = True 2023-11-25T05:02:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: osd objectstore = bluestore 2023-11-25T05:02:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: osd op queue = debug_random 2023-11-25T05:02:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: osd op queue cut off = debug_random 2023-11-25T05:02:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: osd shutdown pgref assert = True 2023-11-25T05:02:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.671 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr] 2023-11-25T05:02:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-11-25T05:02:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 20 2023-11-25T05:02:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2023-11-25T05:02:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2023-11-25T05:02:46.671 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2023-11-25T05:02:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2023-11-25T05:02:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: mgr/telemetry/nag = false 2023-11-25T05:02:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.672 INFO:teuthology.orchestra.run.smithi007.stdout:[mon] 2023-11-25T05:02:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-11-25T05:02:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2023-11-25T05:02:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 20 2023-11-25T05:02:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2023-11-25T05:02:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: mon data avail warn = 5 2023-11-25T05:02:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: mon mgr mkfs grace = 240 2023-11-25T05:02:46.672 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2023-11-25T05:02:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd reporter subtree level = osd 2023-11-25T05:02:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd prime pg temp = true 2023-11-25T05:02:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2023-11-25T05:02:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-25T05:02:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: auth mon ticket ttl = 660 # 11m 2023-11-25T05:02:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: auth service ticket ttl = 240 # 4m 2023-11-25T05:02:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.673 INFO:teuthology.orchestra.run.smithi007.stdout: # don't complain about insecure global_id in the test suite 2023-11-25T05:02:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-25T05:02:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-25T05:02:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: # 1m isn't quite enough 2023-11-25T05:02:46.674 INFO:teuthology.orchestra.run.smithi007.stdout: mon_down_mkfs_grace = 2m 2023-11-25T05:02:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_filestore_osds = false 2023-11-25T05:02:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: mon min osdmap epochs = 50 2023-11-25T05:02:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd crush smoke test = False 2023-11-25T05:02:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: mon osdmap full prune interval = 2 2023-11-25T05:02:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: mon osdmap full prune min = 15 2023-11-25T05:02:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: mon osdmap full prune txsize = 2 2023-11-25T05:02:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: mon scrub interval = 300 2023-11-25T05:02:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on pool no app = False 2023-11-25T05:02:46.676 INFO:teuthology.orchestra.run.smithi007.stdout: paxos service trim min = 10 2023-11-25T05:02:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:02:46.677 INFO:teuthology.orchestra.run.smithi007.stdout:[client] 2023-11-25T05:02:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: rgw cache enabled = true 2023-11-25T05:02:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable ops log = true 2023-11-25T05:02:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable usage log = true 2023-11-25T05:02:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-25T05:02:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-25T05:02:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: debug monc = 20 2023-11-25T05:02:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-11-25T05:02:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: debug objecter = 20 2023-11-25T05:02:46.677 INFO:teuthology.orchestra.run.smithi007.stdout: debug rados = 20 2023-11-25T05:02:46.677 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.a] 2023-11-25T05:02:46.677 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.c] 2023-11-25T05:02:46.678 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.b] 2023-11-25T05:02:46.679 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-25T05:02:46.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-25T05:02:46.779 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-25T05:02:46.779 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:02:46.780 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-25T05:02:46.795 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:02:46.796 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-25T05:02:46.850 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi007.front.sepia.ceph.com 2023-11-25T05:02:46.850 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:02:46.850 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-25T05:02:46.851 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-25T05:02:46.937 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:02:46.937 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-25T05:02:46.992 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi090.front.sepia.ceph.com 2023-11-25T05:02:46.992 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:02:46.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-25T05:02:46.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-25T05:02:47.032 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:02:47.032 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-25T05:02:47.087 INFO:tasks.ceph:Setting up mon nodes... 2023-11-25T05:02:47.087 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-25T05:02:47.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-11-25T05:02:47.150 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-11-25T05:02:47.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-25T05:02:47.218 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-25T05:02:47.220 INFO:tasks.ceph:Setting up mds nodes... 2023-11-25T05:02:47.220 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-25T05:02:47.220 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-25T05:02:47.267 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-25T05:02:47.285 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-11-25T05:02:47.328 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-11-25T05:02:47.343 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-25T05:02:47.343 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-25T05:02:47.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-25T05:02:47.374 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-25T05:02:47.374 INFO:tasks.ceph:role: osd.0 2023-11-25T05:02:47.374 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-25T05:02:47.374 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-25T05:02:47.444 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-25T05:02:47.444 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-25T05:02:47.445 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-25T05:02:47.445 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-25T05:02:47.445 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-25T05:02:47.445 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-11-25T05:02:47.445 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-25T05:02:47.445 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-25T05:02:47.445 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-25T05:02:47.445 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-25T05:02:47.477 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-11-25T05:02:47.479 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-11-25T05:02:47.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-25T05:02:47.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-25T05:02:47.635 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-25T05:02:47.706 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-25T05:02:47.706 INFO:tasks.ceph:role: osd.1 2023-11-25T05:02:47.706 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-25T05:02:47.706 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-25T05:02:47.776 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-25T05:02:47.777 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-25T05:02:47.777 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-25T05:02:47.777 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-25T05:02:47.777 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-25T05:02:47.777 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-11-25T05:02:47.777 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-25T05:02:47.778 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-25T05:02:47.778 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-25T05:02:47.778 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-25T05:02:47.809 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-11-25T05:02:47.811 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-11-25T05:02:47.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-25T05:02:47.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-25T05:02:47.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-25T05:02:48.032 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-25T05:02:48.032 INFO:tasks.ceph:role: osd.2 2023-11-25T05:02:48.032 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-25T05:02:48.033 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-25T05:02:48.102 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-25T05:02:48.102 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-25T05:02:48.102 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-25T05:02:48.103 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-25T05:02:48.103 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-25T05:02:48.103 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-11-25T05:02:48.103 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-25T05:02:48.103 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-25T05:02:48.103 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-25T05:02:48.103 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-25T05:02:48.135 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-11-25T05:02:48.137 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-11-25T05:02:48.137 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-25T05:02:48.212 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-25T05:02:48.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-11-25T05:02:48.358 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-25T05:02:48.358 INFO:tasks.ceph:role: osd.3 2023-11-25T05:02:48.358 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-25T05:02:48.358 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-25T05:02:48.428 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-25T05:02:48.428 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-25T05:02:48.428 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-25T05:02:48.428 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-25T05:02:48.429 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-25T05:02:48.429 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-11-25T05:02:48.429 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-25T05:02:48.429 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-25T05:02:48.429 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-25T05:02:48.429 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-25T05:02:48.460 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-11-25T05:02:48.462 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-11-25T05:02:48.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-11-25T05:02:48.536 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-11-25T05:02:48.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-25T05:02:48.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:48.713+0000 7f81ed2cf500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-11-25T05:02:48.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:48.713+0000 7f81ed2cf500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-11-25T05:02:48.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:48.713+0000 7f81ed2cf500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-25T05:02:48.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:48.713+0000 7f81ed2cf500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-25T05:02:48.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:48.713+0000 7f81ed2cf500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-25T05:02:48.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:48.713+0000 7f81ed2cf500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-11-25T05:02:51.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-25T05:02:51.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-25T05:02:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:51.693+0000 7f5753c12500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-11-25T05:02:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:51.694+0000 7f5753c12500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-11-25T05:02:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:51.694+0000 7f5753c12500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-25T05:02:51.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:51.694+0000 7f5753c12500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-25T05:02:51.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:51.694+0000 7f5753c12500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-25T05:02:51.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:51.694+0000 7f5753c12500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-11-25T05:02:54.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-25T05:02:54.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-25T05:02:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:54.668+0000 7f0224e0e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-11-25T05:02:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:54.668+0000 7f0224e0e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-11-25T05:02:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:54.668+0000 7f0224e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-25T05:02:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:54.668+0000 7f0224e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-25T05:02:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:54.668+0000 7f0224e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-25T05:02:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:54.669+0000 7f0224e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-11-25T05:02:57.513 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-25T05:02:57.549 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-25T05:02:57.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:57.643+0000 7f92eed1a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-11-25T05:02:57.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:57.643+0000 7f92eed1a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-11-25T05:02:57.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:57.643+0000 7f92eed1a500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-25T05:02:57.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:57.643+0000 7f92eed1a500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-25T05:02:57.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:57.643+0000 7f92eed1a500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-25T05:02:57.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:02:57.644+0000 7f92eed1a500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-11-25T05:03:00.494 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-11-25T05:03:00.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-11-25T05:03:00.563 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-25T05:03:00.563 INFO:tasks.ceph:role: osd.4 2023-11-25T05:03:00.563 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-25T05:03:00.563 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-25T05:03:00.642 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-25T05:03:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-25T05:03:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-25T05:03:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-25T05:03:00.643 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-25T05:03:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-11-25T05:03:00.643 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-25T05:03:00.643 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-25T05:03:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-25T05:03:00.643 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-25T05:03:01.161 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2023-11-25T05:03:01.164 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-11-25T05:03:01.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-11-25T05:03:01.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-11-25T05:03:01.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-11-25T05:03:01.406 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-25T05:03:01.407 INFO:tasks.ceph:role: osd.5 2023-11-25T05:03:01.407 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-25T05:03:01.407 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-25T05:03:01.483 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-25T05:03:01.483 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-25T05:03:01.483 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-25T05:03:01.483 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-25T05:03:01.484 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-25T05:03:01.484 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-11-25T05:03:01.484 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-25T05:03:01.484 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-25T05:03:01.484 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-25T05:03:01.484 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-25T05:03:01.938 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2023-11-25T05:03:01.940 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-11-25T05:03:01.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-11-25T05:03:02.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-11-25T05:03:02.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-11-25T05:03:02.147 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-25T05:03:02.147 INFO:tasks.ceph:role: osd.6 2023-11-25T05:03:02.147 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-25T05:03:02.147 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-25T05:03:02.222 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-25T05:03:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-25T05:03:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-25T05:03:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-25T05:03:02.223 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-25T05:03:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-11-25T05:03:02.223 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-25T05:03:02.223 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-25T05:03:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-25T05:03:02.223 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-25T05:03:02.706 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2023-11-25T05:03:02.708 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-11-25T05:03:02.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-11-25T05:03:02.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-11-25T05:03:02.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-11-25T05:03:02.967 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-25T05:03:02.967 INFO:tasks.ceph:role: osd.7 2023-11-25T05:03:02.967 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-25T05:03:02.967 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-25T05:03:03.042 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-25T05:03:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-25T05:03:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-25T05:03:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-25T05:03:03.043 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-25T05:03:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-11-25T05:03:03.043 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-25T05:03:03.043 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-25T05:03:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-25T05:03:03.044 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-25T05:03:03.562 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2023-11-25T05:03:03.564 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-11-25T05:03:03.565 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-11-25T05:03:03.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-11-25T05:03:03.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-25T05:03:03.833 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:03.830+0000 7fd401e13500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-11-25T05:03:03.834 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:03.831+0000 7fd401e13500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-11-25T05:03:03.834 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:03.831+0000 7fd401e13500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-25T05:03:03.834 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:03.831+0000 7fd401e13500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-25T05:03:03.834 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:03.831+0000 7fd401e13500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-25T05:03:03.834 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:03.831+0000 7fd401e13500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-11-25T05:03:06.676 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-11-25T05:03:06.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-25T05:03:06.801 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:06.799+0000 7fb95f907500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-11-25T05:03:06.801 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:06.800+0000 7fb95f907500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-11-25T05:03:06.802 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:06.800+0000 7fb95f907500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-25T05:03:06.802 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:06.800+0000 7fb95f907500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-25T05:03:06.802 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:06.800+0000 7fb95f907500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-25T05:03:06.802 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:06.800+0000 7fb95f907500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-11-25T05:03:09.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-11-25T05:03:09.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-25T05:03:09.777 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:09.775+0000 7fa34320f500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-11-25T05:03:09.777 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:09.775+0000 7fa34320f500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-11-25T05:03:09.777 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:09.775+0000 7fa34320f500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-25T05:03:09.777 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:09.775+0000 7fa34320f500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-25T05:03:09.778 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:09.775+0000 7fa34320f500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-25T05:03:09.778 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:09.776+0000 7fa34320f500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-11-25T05:03:12.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-11-25T05:03:12.665 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-25T05:03:12.759 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:12.755+0000 7f6bd8a49500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-11-25T05:03:12.760 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:12.755+0000 7f6bd8a49500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-11-25T05:03:12.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:12.755+0000 7f6bd8a49500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-25T05:03:12.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:12.755+0000 7f6bd8a49500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-25T05:03:12.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:12.755+0000 7f6bd8a49500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-25T05:03:12.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:12.756+0000 7f6bd8a49500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-11-25T05:03:15.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-11-25T05:03:15.631 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-25T05:03:15.631 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:03:15.632 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-11-25T05:03:15.668 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:03:15.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-25T05:03:15.743 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:03:15.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-25T05:03:15.817 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:03:15.817 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-25T05:03:15.890 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:03:15.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-11-25T05:03:15.963 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:03:15.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-25T05:03:15.995 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:03:15.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-11-25T05:03:16.065 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:03:16.065 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-11-25T05:03:16.132 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:03:16.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-11-25T05:03:16.199 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:03:16.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-11-25T05:03:16.266 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:03:16.266 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-25T05:03:16.283 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:03:16.283 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-11-25T05:03:16.321 INFO:tasks.ceph:Adding keys to all mons... 2023-11-25T05:03:16.321 DEBUG:teuthology.orchestra.run.smithi007:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-25T05:03:16.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-25T05:03:16.364 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.y] 2023-11-25T05:03:16.364 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD3f2Fl/7zxCBAAN3bZ83RKZaWCQr1h8ujEPw== 2023-11-25T05:03:16.365 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2023-11-25T05:03:16.365 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD4f2FlL3KPKhAAgkhWdwlv7oxa25/pdvg28Q== 2023-11-25T05:03:16.365 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2023-11-25T05:03:16.365 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD7f2FlX3xqKRAAwcmKhCO+P7SL4d9vnow7pQ== 2023-11-25T05:03:16.365 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2023-11-25T05:03:16.365 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD+f2Fl2trnJxAAdwjzAmY5UQdShnRR7lZ0eQ== 2023-11-25T05:03:16.365 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.3] 2023-11-25T05:03:16.365 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQABgGFlxAJqJhAA1tNY7hLbF+O/QY8WsC2ylQ== 2023-11-25T05:03:16.365 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.x] 2023-11-25T05:03:16.365 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD3f2Flrzz3DBAACy2dw8txGYpfPWQFxSI76g== 2023-11-25T05:03:16.365 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.4] 2023-11-25T05:03:16.365 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAHgGFlsE6lMRAAx7ZNMQ12U82DPV11ZQg8Hw== 2023-11-25T05:03:16.366 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.5] 2023-11-25T05:03:16.366 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAKgGFlRHm7LxAAbP+GtUl3LNperqnvVpqqIw== 2023-11-25T05:03:16.366 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.6] 2023-11-25T05:03:16.366 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQANgGFlWFtKLhAAOIaY6EevkV2IlzxolULVaQ== 2023-11-25T05:03:16.366 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.7] 2023-11-25T05:03:16.366 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAQgGFlaTMWLRAAZoKLtGDggcz0OCHHVfmxTg== 2023-11-25T05:03:16.366 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2023-11-25T05:03:16.366 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD3f2FlPJ/qDxAA17xEooFbgLC6yvaWmYDccQ== 2023-11-25T05:03:16.366 INFO:teuthology.orchestra.run.smithi007.stdout:[client.1] 2023-11-25T05:03:16.366 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD3f2FldOWJExAA2tSdIiK5ZIQMJh1wBuIzug== 2023-11-25T05:03:16.387 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr.y] 2023-11-25T05:03:16.387 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQD3f2Fl/7zxCBAAN3bZ83RKZaWCQr1h8ujEPw== 2023-11-25T05:03:16.387 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.0] 2023-11-25T05:03:16.387 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQD4f2FlL3KPKhAAgkhWdwlv7oxa25/pdvg28Q== 2023-11-25T05:03:16.387 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.1] 2023-11-25T05:03:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQD7f2FlX3xqKRAAwcmKhCO+P7SL4d9vnow7pQ== 2023-11-25T05:03:16.388 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.2] 2023-11-25T05:03:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQD+f2Fl2trnJxAAdwjzAmY5UQdShnRR7lZ0eQ== 2023-11-25T05:03:16.388 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.3] 2023-11-25T05:03:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQABgGFlxAJqJhAA1tNY7hLbF+O/QY8WsC2ylQ== 2023-11-25T05:03:16.388 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr.x] 2023-11-25T05:03:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQD3f2Flrzz3DBAACy2dw8txGYpfPWQFxSI76g== 2023-11-25T05:03:16.388 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.4] 2023-11-25T05:03:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQAHgGFlsE6lMRAAx7ZNMQ12U82DPV11ZQg8Hw== 2023-11-25T05:03:16.388 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.5] 2023-11-25T05:03:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQAKgGFlRHm7LxAAbP+GtUl3LNperqnvVpqqIw== 2023-11-25T05:03:16.388 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.6] 2023-11-25T05:03:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQANgGFlWFtKLhAAOIaY6EevkV2IlzxolULVaQ== 2023-11-25T05:03:16.389 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.7] 2023-11-25T05:03:16.389 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQAQgGFlaTMWLRAAZoKLtGDggcz0OCHHVfmxTg== 2023-11-25T05:03:16.389 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2023-11-25T05:03:16.389 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQD3f2FlPJ/qDxAA17xEooFbgLC6yvaWmYDccQ== 2023-11-25T05:03:16.389 INFO:teuthology.orchestra.run.smithi090.stdout:[client.1] 2023-11-25T05:03:16.389 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQD3f2FldOWJExAA2tSdIiK5ZIQMJh1wBuIzug== 2023-11-25T05:03:16.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-25T05:03:16.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-25T05:03:16.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.485 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.543 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.708 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-25T05:03:16.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-25T05:03:16.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.819 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.875 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.877 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-25T05:03:16.987 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-25T05:03:16.989 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-25T05:03:17.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-25T05:03:17.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-25T05:03:17.104 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-25T05:03:17.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-25T05:03:17.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-25T05:03:17.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-25T05:03:17.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-11-25T05:03:17.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-25T05:03:17.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-11-25T05:03:17.469 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-11-25T05:03:17.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-25T05:03:17.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-11-25T05:03:17.626 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-25T05:03:17.628 DEBUG:teuthology.orchestra.run.smithi090:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-25T05:03:17.681 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-25T05:03:17.681 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-11-25T05:03:17.681 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-25T05:03:17.681 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-11-25T05:03:17.683 INFO:tasks.ceph.mon.a:Started 2023-11-25T05:03:17.684 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-11-25T05:03:17.684 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-25T05:03:17.684 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-11-25T05:03:17.686 INFO:tasks.ceph.mon.c:Started 2023-11-25T05:03:17.686 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-11-25T05:03:17.686 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-25T05:03:17.686 DEBUG:teuthology.orchestra.run.smithi090:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-11-25T05:03:17.725 INFO:tasks.ceph.mon.b:Started 2023-11-25T05:03:17.726 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-25T05:03:17.726 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-11-25T05:03:17.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-11-25T05:03:17.728 INFO:tasks.ceph.mgr.y:Started 2023-11-25T05:03:17.728 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-25T05:03:17.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-11-25T05:03:17.730 INFO:tasks.ceph.mgr.x:Started 2023-11-25T05:03:17.731 INFO:tasks.ceph:Setting crush tunables to default 2023-11-25T05:03:17.731 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd crush tunables default 2023-11-25T05:03:17.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.835+0000 7f62ce6f2640 10 monclient: get_monmap_and_config 2023-11-25T05:03:17.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.835+0000 7f62ce6f2640 10 monclient: build_initial_monmap 2023-11-25T05:03:17.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.835+0000 7f62ce6f2640 10 monclient: monmap: 2023-11-25T05:03:17.838 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:03:17.838 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:17.838 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:03:17.837417+0000 2023-11-25T05:03:17.838 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:03:17.837417+0000 2023-11-25T05:03:17.838 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:17.838 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:17.838 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:03:17.838 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:03:17.839 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:03:17.839 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:03:17.839 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:03:17.839 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:03:17.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.836+0000 7f62ce6f2640 1 Processor -- start 2023-11-25T05:03:17.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.836+0000 7f62ce6f2640 1 -- start start 2023-11-25T05:03:17.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.836+0000 7f62ce6f2640 10 monclient: init 2023-11-25T05:03:17.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.836+0000 7f62ce6f2640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:17.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.836+0000 7f62ce6f2640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-11-25T05:03:17.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.836+0000 7f62ce6f2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62c80a8a00 addr v1:172.21.15.90:6789/0 2023-11-25T05:03:17.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.837+0000 7f62ce6f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62c80b2260 addr v1:172.21.15.7:6789/0 2023-11-25T05:03:17.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.837+0000 7f62ce6f2640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62c80a8de0 0x7f62c813f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:17.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.837+0000 7f62ce6f2640 10 monclient(hunting): picked mon.noname-c con 0x7f62c80a8de0 addr v2:172.21.15.90:3300/0 2023-11-25T05:03:17.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.837+0000 7f62ce6f2640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c80aa970 con 0x7f62c80b2260 2023-11-25T05:03:17.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.837+0000 7f62ce6f2640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c80aac50 con 0x7f62c80a8a00 2023-11-25T05:03:17.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.837+0000 7f62ce6f2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:17.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.837+0000 7f62ce6f2640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c81421b0 con 0x7f62c80a8de0 2023-11-25T05:03:17.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.837+0000 7f62ce6f2640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:17.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.837+0000 7f62ce6f2640 10 monclient(hunting): authenticate will time out at 1324.080962s 2023-11-25T05:03:17.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.837+0000 7f62c77fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f62c80b2260 legacy=0x7f62c813f080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-25T05:03:17.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.837+0000 7f62ccc8f640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62c80a8de0 msgr2=0x7f62c813f790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2023-11-25T05:03:17.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.837+0000 7f62c7fff640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62c80a8a00 legacy=0x7f62c81425d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2023-11-25T05:03:17.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:17.837+0000 7f62ccc8f640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62c80a8de0 0x7f62c813f790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-25T05:03:18.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:18.038+0000 7f62c77fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f62c80b2260 legacy=0x7f62c813f080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-25T05:03:18.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:18.038+0000 7f62ccc8f640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62c80a8de0 msgr2=0x7f62c813f790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2023-11-25T05:03:18.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:18.038+0000 7f62c7fff640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62c80a8a00 legacy=0x7f62c81425d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2023-11-25T05:03:18.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:18.038+0000 7f62ccc8f640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62c80a8de0 0x7f62c813f790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-25T05:03:18.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:18.439+0000 7f62c77fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f62c80b2260 legacy=0x7f62c813f080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-25T05:03:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:18.439+0000 7f62c7fff640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62c80a8a00 legacy=0x7f62c81425d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2023-11-25T05:03:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:18.439+0000 7f62ccc8f640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62c80a8de0 msgr2=0x7f62c813f790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2023-11-25T05:03:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:18.439+0000 7f62ccc8f640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62c80a8de0 0x7f62c813f790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-25T05:03:19.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:19.240+0000 7f62c77fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f62c80b2260 legacy=0x7f62c813f080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-25T05:03:19.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:19.240+0000 7f62c7fff640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62c80a8a00 legacy=0x7f62c81425d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2023-11-25T05:03:19.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:19.240+0000 7f62ccc8f640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62c80a8de0 msgr2=0x7f62c813f790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2023-11-25T05:03:19.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:19.240+0000 7f62ccc8f640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62c80a8de0 0x7f62c813f790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-25T05:03:20.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 10 monclient(hunting): tick 2023-11-25T05:03:20.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 10 monclient(hunting): _check_auth_tickets 2023-11-25T05:03:20.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 1 monclient(hunting): continuing hunt 2023-11-25T05:03:20.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 10 monclient(hunting): _reopen_session rank -1 2023-11-25T05:03:20.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62c80a8de0 msgr2=0x7f62c813f790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-25T05:03:20.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62c80a8de0 0x7f62c813f790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:20.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62c80a8a00 legacy=0x7f62c81425d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-25T05:03:20.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f62c80b2260 legacy=0x7f62c813f080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-25T05:03:20.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 10 monclient: _add_conns ranks=[3,2] 2023-11-25T05:03:20.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f62b40036a0 0x7f62b4005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f62b40036a0 addr v2:172.21.15.7:3300/0 2023-11-25T05:03:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 0x7f62b4008670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f62b40061b0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c81421b0 con 0x7f62b40036a0 2023-11-25T05:03:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b4008d50 con 0x7f62b40061b0 2023-11-25T05:03:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c5ffb640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c77fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 msgr2=0x7f62b4008670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2023-11-25T05:03:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c7fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f62b40036a0 msgr2=0x7f62b4005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-25T05:03:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c77fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 0x7f62b4008670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-25T05:03:20.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:20.837+0000 7f62c7fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f62b40036a0 0x7f62b4005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-25T05:03:21.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:21.038+0000 7f62c7fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f62b40036a0 msgr2=0x7f62b4005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-25T05:03:21.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:21.038+0000 7f62c77fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 msgr2=0x7f62b4008670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2023-11-25T05:03:21.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:21.038+0000 7f62c7fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f62b40036a0 0x7f62b4005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-25T05:03:21.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:21.038+0000 7f62c77fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 0x7f62b4008670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-25T05:03:21.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:21.439+0000 7f62c7fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f62b40036a0 msgr2=0x7f62b4005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-25T05:03:21.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:21.439+0000 7f62c77fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 msgr2=0x7f62b4008670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2023-11-25T05:03:21.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:21.439+0000 7f62c7fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f62b40036a0 0x7f62b4005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-25T05:03:21.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:21.439+0000 7f62c77fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 0x7f62b4008670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-25T05:03:22.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:22.240+0000 7f62c7fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f62b40036a0 msgr2=0x7f62b4005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-25T05:03:22.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:22.240+0000 7f62c77fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 msgr2=0x7f62b4008670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2023-11-25T05:03:22.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:22.240+0000 7f62c7fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f62b40036a0 0x7f62b4005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-25T05:03:22.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:22.240+0000 7f62c77fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 0x7f62b4008670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-25T05:03:23.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 10 monclient(hunting): tick 2023-11-25T05:03:23.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 10 monclient(hunting): _check_auth_tickets 2023-11-25T05:03:23.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 1 monclient(hunting): continuing hunt 2023-11-25T05:03:23.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 10 monclient(hunting): _reopen_session rank -1 2023-11-25T05:03:23.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 msgr2=0x7f62b4008670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-25T05:03:23.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 0x7f62b4008670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:23.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f62b40036a0 msgr2=0x7f62b4005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-25T05:03:23.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f62b40036a0 0x7f62b4005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:23.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-11-25T05:03:23.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62ccc8f640 1 -- reap_dead start 2023-11-25T05:03:23.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 0x7f62b4008670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:23.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f62b40061b0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:23.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62c80b2260 addr v1:172.21.15.7:6789/0 2023-11-25T05:03:23.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62b4003b10 0x7f62b400c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:23.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f62b4003b10 addr v2:172.21.15.90:3300/0 2023-11-25T05:03:23.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c7fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 msgr2=0x7f62b4008670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2023-11-25T05:03:23.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c80aaf30 con 0x7f62c80b2260 2023-11-25T05:03:23.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:23.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c81421b0 con 0x7f62b4003b10 2023-11-25T05:03:23.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:23.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b4008d50 con 0x7f62b40061b0 2023-11-25T05:03:23.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c77fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f62c80b2260 legacy=0x7f62b400afe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-25T05:03:23.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c5ffb640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:23.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62c7fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 0x7f62b4008670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-25T05:03:23.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62ccc8f640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62b4003b10 msgr2=0x7f62b400c750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2023-11-25T05:03:23.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:23.838+0000 7f62ccc8f640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62b4003b10 0x7f62b400c750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-25T05:03:24.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:24.039+0000 7f62c77fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f62c80b2260 legacy=0x7f62b400afe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-25T05:03:24.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:24.039+0000 7f62c7fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 msgr2=0x7f62b4008670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2023-11-25T05:03:24.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:24.039+0000 7f62c7fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 0x7f62b4008670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-25T05:03:24.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:24.039+0000 7f62ccc8f640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62b4003b10 msgr2=0x7f62b400c750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2023-11-25T05:03:24.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:24.039+0000 7f62ccc8f640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62b4003b10 0x7f62b400c750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-25T05:03:24.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:24.440+0000 7f62c77fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f62c80b2260 legacy=0x7f62b400afe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-25T05:03:24.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:24.440+0000 7f62c7fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 msgr2=0x7f62b4008670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2023-11-25T05:03:24.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:24.440+0000 7f62ccc8f640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62b4003b10 msgr2=0x7f62b400c750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2023-11-25T05:03:24.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:24.440+0000 7f62c7fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 0x7f62b4008670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-25T05:03:24.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:24.440+0000 7f62ccc8f640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62b4003b10 0x7f62b400c750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-25T05:03:25.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:25.241+0000 7f62c77fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f62c80b2260 legacy=0x7f62b400afe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2023-11-25T05:03:25.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:25.241+0000 7f62c7fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 msgr2=0x7f62b4008670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2023-11-25T05:03:25.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:25.241+0000 7f62c7fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 0x7f62b4008670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-25T05:03:25.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:25.241+0000 7f62ccc8f640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62b4003b10 msgr2=0x7f62b400c750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2023-11-25T05:03:25.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:25.241+0000 7f62ccc8f640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62b4003b10 0x7f62b400c750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-25T05:03:25.284 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:00:07.428 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:03:25.598 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:00:07.740 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:03:25.882 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:00:08.081 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:03:25.884 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:00:08.083 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:03:26.215 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:00:08.413 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:03:26.217 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:00:08.415 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:03:26.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 10 monclient(hunting): tick 2023-11-25T05:03:26.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 10 monclient(hunting): _check_auth_tickets 2023-11-25T05:03:26.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 1 monclient(hunting): continuing hunt 2023-11-25T05:03:26.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 10 monclient(hunting): _reopen_session rank -1 2023-11-25T05:03:26.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 msgr2=0x7f62b4008670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-25T05:03:26.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b40061b0 0x7f62b4008670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:26.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62b4003b10 msgr2=0x7f62b400c750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-25T05:03:26.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62b4003b10 0x7f62b400c750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:26.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f62c80b2260 legacy=0x7f62b400afe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-25T05:03:26.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 10 monclient: _add_conns ranks=[1,3] 2023-11-25T05:03:26.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62b400cce0 addr v1:172.21.15.90:6789/0 2023-11-25T05:03:26.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:26.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f62b4010d40 addr v2:172.21.15.7:3300/0 2023-11-25T05:03:26.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c80aa970 con 0x7f62b400cce0 2023-11-25T05:03:26.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:26.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c81421b0 con 0x7f62b4010d40 2023-11-25T05:03:26.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c5ffb640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:26.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c77fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 msgr2=0x7f62b4013130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-25T05:03:26.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:26.838+0000 7f62c77fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-25T05:03:27.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:27.039+0000 7f62c77fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 msgr2=0x7f62b4013130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-25T05:03:27.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:27.039+0000 7f62c77fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-25T05:03:27.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:27.070+0000 7f62c7fff640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f62b400cce0 0x7f62b400d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:44342/0 (socket says 172.21.15.7:44342) 2023-11-25T05:03:27.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:27.070+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 learned_addr learned my addr 172.21.15.7:0/2834946697 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:03:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:27.257+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v1:172.21.15.90:6789/0 conn(0x7f62b400cce0 legacy=0x7f62b400d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-11-25T05:03:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:27.257+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v1:172.21.15.90:6789/0 conn(0x7f62b400cce0 legacy=0x7f62b400d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-25T05:03:27.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:27.257+0000 7f62c7fff640 1 --1- 172.21.15.7:0/2834946697 >> v1:172.21.15.90:6789/0 conn(0x7f62b400cce0 0x7f62b400d0f0 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2023-11-25T05:03:27.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:27.257+0000 7f62c7fff640 1 --1- 172.21.15.7:0/2834946697 >> v1:172.21.15.90:6789/0 conn(0x7f62b400cce0 0x7f62b400d0f0 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2023-11-25T05:03:27.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:27.257+0000 7f62c6ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.90:6789/0 2023-11-25T05:03:27.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:27.440+0000 7f62c77fe640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 msgr2=0x7f62b4013130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2023-11-25T05:03:27.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:27.440+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-25T05:03:28.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:28.288+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:28.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:28.289+0000 7f62c77fe640 10 monclient(hunting): get_auth_request con 0x7f62b4010d40 auth_method 0 2023-11-25T05:03:28.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:28.289+0000 7f62c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:28.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:28.289+0000 7f62c77fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:28.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:28.289+0000 7f62c77fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:28.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:28.289+0000 7f62c77fe640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 msgr2=0x7f62b4013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-25T05:03:28.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:28.289+0000 7f62c77fe640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 msgr2=0x7f62b4013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:28.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:28.289+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:28.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:28.289+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-25T05:03:29.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 10 monclient(hunting): tick 2023-11-25T05:03:29.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 10 monclient(hunting): _check_auth_tickets 2023-11-25T05:03:29.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 1 monclient(hunting): continuing hunt 2023-11-25T05:03:29.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 10 monclient(hunting): _reopen_session rank -1 2023-11-25T05:03:29.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 msgr2=0x7f62b4013130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-25T05:03:29.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:29.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 1 -- 172.21.15.7:0/2834946697 >> v1:172.21.15.90:6789/0 conn(0x7f62b400cce0 legacy=0x7f62b400d0f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-25T05:03:29.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-11-25T05:03:29.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:29.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62ccc8f640 1 -- 172.21.15.7:0/2834946697 reap_dead start 2023-11-25T05:03:29.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f62b4010d40 addr v2:172.21.15.7:3300/0 2023-11-25T05:03:29.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62b400c750 addr v1:172.21.15.90:6789/0 2023-11-25T05:03:29.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 0x7f62b4008fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:29.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f62b4006660 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:29.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 1 -- 172.21.15.7:0/2834946697 --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c80aaf30 con 0x7f62b400c750 2023-11-25T05:03:29.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:29.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 1 -- 172.21.15.7:0/2834946697 --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c81421b0 con 0x7f62b4010d40 2023-11-25T05:03:29.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:29.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 1 -- 172.21.15.7:0/2834946697 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b4008d50 con 0x7f62b4006660 2023-11-25T05:03:29.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.838+0000 7f62c5ffb640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:29.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.839+0000 7f62ccc8f640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 0x7f62b4008fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:29.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.839+0000 7f62ccc8f640 10 monclient(hunting): get_auth_request con 0x7f62b4006660 auth_method 0 2023-11-25T05:03:29.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.839+0000 7f62ccc8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:29.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.839+0000 7f62ccc8f640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:29.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.839+0000 7f62ccc8f640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:29.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.849+0000 7f62ccc8f640 1 -- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 msgr2=0x7f62b4008fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-25T05:03:29.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.849+0000 7f62ccc8f640 1 -- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 msgr2=0x7f62b4008fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:29.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.849+0000 7f62ccc8f640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 0x7f62b4008fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:29.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:29.849+0000 7f62ccc8f640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 0x7f62b4008fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-25T05:03:30.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.057+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:30.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.058+0000 7f62c7fff640 10 monclient(hunting): get_auth_request con 0x7f62b4010d40 auth_method 0 2023-11-25T05:03:30.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.058+0000 7f62c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:30.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.058+0000 7f62c7fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:30.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.058+0000 7f62c7fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.063+0000 7f62ccc8f640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 0x7f62b4008fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.064+0000 7f62ccc8f640 10 monclient(hunting): get_auth_request con 0x7f62b4006660 auth_method 0 2023-11-25T05:03:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.064+0000 7f62ccc8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.064+0000 7f62ccc8f640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.064+0000 7f62ccc8f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.065+0000 7f62ccc8f640 1 -- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 msgr2=0x7f62b4008fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-25T05:03:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.065+0000 7f62ccc8f640 1 -- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 msgr2=0x7f62b4008fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:30.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.065+0000 7f62ccc8f640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 0x7f62b4008fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:30.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.065+0000 7f62ccc8f640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 0x7f62b4008fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-25T05:03:30.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.085+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 msgr2=0x7f62b4013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-25T05:03:30.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.085+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 msgr2=0x7f62b4013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:30.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.085+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:30.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.085+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-25T05:03:30.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.286+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:30.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.286+0000 7f62c7fff640 10 monclient(hunting): get_auth_request con 0x7f62b4010d40 auth_method 0 2023-11-25T05:03:30.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.286+0000 7f62c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:30.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.286+0000 7f62c7fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:30.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.286+0000 7f62c7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.296+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 msgr2=0x7f62b4013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-25T05:03:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.296+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 msgr2=0x7f62b4013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.296+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:30.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.296+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-25T05:03:30.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.466+0000 7f62ccc8f640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 0x7f62b4008fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:30.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.466+0000 7f62ccc8f640 10 monclient(hunting): get_auth_request con 0x7f62b4006660 auth_method 0 2023-11-25T05:03:30.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.466+0000 7f62ccc8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:30.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.466+0000 7f62ccc8f640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:30.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.466+0000 7f62ccc8f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:30.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.467+0000 7f62ccc8f640 1 -- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 msgr2=0x7f62b4008fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-25T05:03:30.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.467+0000 7f62ccc8f640 1 -- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 msgr2=0x7f62b4008fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:30.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.467+0000 7f62ccc8f640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 0x7f62b4008fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:30.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.467+0000 7f62ccc8f640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 0x7f62b4008fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-25T05:03:30.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.697+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:30.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.698+0000 7f62c7fff640 10 monclient(hunting): get_auth_request con 0x7f62b4010d40 auth_method 0 2023-11-25T05:03:30.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.698+0000 7f62c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:30.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.698+0000 7f62c7fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:30.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.698+0000 7f62c7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:30.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.698+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 msgr2=0x7f62b4013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-25T05:03:30.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.698+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 msgr2=0x7f62b4013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:30.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.698+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:30.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:30.698+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-25T05:03:31.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.268+0000 7f62ccc8f640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 0x7f62b4008fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:31.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.269+0000 7f62ccc8f640 10 monclient(hunting): get_auth_request con 0x7f62b4006660 auth_method 0 2023-11-25T05:03:31.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.269+0000 7f62ccc8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:31.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.269+0000 7f62ccc8f640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:31.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.269+0000 7f62ccc8f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:31.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.270+0000 7f62ccc8f640 1 -- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 msgr2=0x7f62b4008fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-25T05:03:31.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.270+0000 7f62ccc8f640 1 -- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 msgr2=0x7f62b4008fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:31.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.270+0000 7f62ccc8f640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 0x7f62b4008fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:31.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.270+0000 7f62ccc8f640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 0x7f62b4008fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-25T05:03:31.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.500+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:31.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.500+0000 7f62c7fff640 10 monclient(hunting): get_auth_request con 0x7f62b4010d40 auth_method 0 2023-11-25T05:03:31.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.500+0000 7f62c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:31.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.500+0000 7f62c7fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:31.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.500+0000 7f62c7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:31.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.501+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 msgr2=0x7f62b4013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-25T05:03:31.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.501+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 msgr2=0x7f62b4013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:31.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.501+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:31.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:31.501+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-25T05:03:32.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 10 monclient(hunting): tick 2023-11-25T05:03:32.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 10 monclient(hunting): _check_auth_tickets 2023-11-25T05:03:32.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 1 monclient(hunting): continuing hunt 2023-11-25T05:03:32.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 10 monclient(hunting): _reopen_session rank -1 2023-11-25T05:03:32.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 1 -- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 msgr2=0x7f62b4008fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-25T05:03:32.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b4006660 0x7f62b4008fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:32.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 msgr2=0x7f62b4013130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-25T05:03:32.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.7:3300/0 conn(0x7f62b4010d40 0x7f62b4013130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:32.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 1 -- 172.21.15.7:0/2834946697 >> v1:172.21.15.90:6789/0 conn(0x7f62b400c750 legacy=0x7f62b400d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:32.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 10 monclient: _add_conns ranks=[0,4] 2023-11-25T05:03:32.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62b4009510 addr v1:172.21.15.7:6789/0 2023-11-25T05:03:32.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:32.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f62b4010440 addr v2:172.21.15.90:3300/0 2023-11-25T05:03:32.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 1 -- 172.21.15.7:0/2834946697 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c80aac50 con 0x7f62b4009510 2023-11-25T05:03:32.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:32.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 1 -- 172.21.15.7:0/2834946697 --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c81421b0 con 0x7f62b4010440 2023-11-25T05:03:32.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.839+0000 7f62c5ffb640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:32.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.840+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:32.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.841+0000 7f62c77fe640 10 monclient(hunting): get_auth_request con 0x7f62b4010440 auth_method 0 2023-11-25T05:03:32.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.841+0000 7f62c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:32.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.841+0000 7f62c77fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:32.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.841+0000 7f62c77fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:32.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.842+0000 7f62c77fe640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-25T05:03:32.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.842+0000 7f62c77fe640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:32.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.842+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:32.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:32.842+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-25T05:03:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.043+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.044+0000 7f62c77fe640 10 monclient(hunting): get_auth_request con 0x7f62b4010440 auth_method 0 2023-11-25T05:03:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.044+0000 7f62c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.044+0000 7f62c77fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.044+0000 7f62c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:33.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.045+0000 7f62c77fe640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-25T05:03:33.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.045+0000 7f62c77fe640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:33.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.045+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:33.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.045+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-25T05:03:33.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.446+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:33.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.447+0000 7f62c77fe640 10 monclient(hunting): get_auth_request con 0x7f62b4010440 auth_method 0 2023-11-25T05:03:33.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.447+0000 7f62c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:33.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.447+0000 7f62c77fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:33.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.447+0000 7f62c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:33.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.448+0000 7f62c77fe640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-25T05:03:33.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.448+0000 7f62c77fe640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:33.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.448+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:33.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:33.448+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-25T05:03:34.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:34.249+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:34.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:34.250+0000 7f62c77fe640 10 monclient(hunting): get_auth_request con 0x7f62b4010440 auth_method 0 2023-11-25T05:03:34.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:34.250+0000 7f62c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:34.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:34.250+0000 7f62c77fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:34.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:34.250+0000 7f62c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:34.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:34.251+0000 7f62c77fe640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-25T05:03:34.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:34.251+0000 7f62c77fe640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:34.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:34.251+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:34.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:34.251+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-25T05:03:35.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 10 monclient(hunting): tick 2023-11-25T05:03:35.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 10 monclient(hunting): _check_auth_tickets 2023-11-25T05:03:35.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 1 monclient(hunting): continuing hunt 2023-11-25T05:03:35.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 10 monclient(hunting): _reopen_session rank -1 2023-11-25T05:03:35.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-25T05:03:35.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:35.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 1 -- 172.21.15.7:0/2834946697 >> v1:172.21.15.7:6789/0 conn(0x7f62b4009510 legacy=0x7f62b400fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:35.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-11-25T05:03:35.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:35.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f62b4010440 addr v2:172.21.15.90:3300/0 2023-11-25T05:03:35.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62ccc8f640 1 -- 172.21.15.7:0/2834946697 reap_dead start 2023-11-25T05:03:35.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62b400ae70 addr v1:172.21.15.7:6789/0 2023-11-25T05:03:35.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62b4006660 addr v1:172.21.15.90:6789/0 2023-11-25T05:03:35.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 1 -- 172.21.15.7:0/2834946697 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b40109c0 con 0x7f62b400ae70 2023-11-25T05:03:35.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 1 -- 172.21.15.7:0/2834946697 --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b400b270 con 0x7f62b4006660 2023-11-25T05:03:35.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:35.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 1 -- 172.21.15.7:0/2834946697 --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c81421b0 con 0x7f62b4010440 2023-11-25T05:03:35.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.839+0000 7f62c5ffb640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:35.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.950+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:35.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.952+0000 7f62c7fff640 10 monclient(hunting): get_auth_request con 0x7f62b4010440 auth_method 0 2023-11-25T05:03:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.952+0000 7f62c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.952+0000 7f62c7fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.952+0000 7f62c7fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:35.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.956+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-25T05:03:35.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.956+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:35.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.956+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:35.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:35.956+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-25T05:03:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.159+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.160+0000 7f62c7fff640 10 monclient(hunting): get_auth_request con 0x7f62b4010440 auth_method 0 2023-11-25T05:03:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.160+0000 7f62c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.160+0000 7f62c7fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.160+0000 7f62c7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.161+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-25T05:03:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.161+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.161+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.161+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-25T05:03:36.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.562+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:36.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.562+0000 7f62c7fff640 10 monclient(hunting): get_auth_request con 0x7f62b4010440 auth_method 0 2023-11-25T05:03:36.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.562+0000 7f62c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:36.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.562+0000 7f62c7fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:36.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.562+0000 7f62c7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:36.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.563+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-25T05:03:36.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.563+0000 7f62c7fff640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-25T05:03:36.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.563+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:03:36.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.563+0000 7f62c7fff640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-25T05:03:36.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.734+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787115843 0 0) 0x7f62b40109c0 con 0x7f62b400ae70 2023-11-25T05:03:36.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.734+0000 7f62c6ffd640 10 monclient(hunting): _negotiate 2023-11-25T05:03:36.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.734+0000 7f62c6ffd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:36.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.734+0000 7f62c6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:36.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.734+0000 7f62c6ffd640 10 monclient(hunting): my global_id is 4101 2023-11-25T05:03:36.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.734+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62b0003520 con 0x7f62b400ae70 2023-11-25T05:03:36.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.770+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852008899 0 0) 0x7f62b400b270 con 0x7f62b4006660 2023-11-25T05:03:36.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.770+0000 7f62c6ffd640 10 monclient(hunting): _negotiate 2023-11-25T05:03:36.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.770+0000 7f62c6ffd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:36.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.770+0000 7f62c6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:36.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.770+0000 7f62c6ffd640 10 monclient(hunting): my global_id is 4099 2023-11-25T05:03:36.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.770+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62b40109c0 con 0x7f62b4006660 2023-11-25T05:03:36.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1920830185 0 0) 0x7f62b0003520 con 0x7f62b400ae70 2023-11-25T05:03:36.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 msgr2=0x7f62b400a930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-25T05:03:36.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 1 --2- 172.21.15.7:0/2834946697 >> v2:172.21.15.90:3300/0 conn(0x7f62b4010440 0x7f62b400a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:36.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 >> v1:172.21.15.90:6789/0 conn(0x7f62b4006660 legacy=0x7f62b400d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:36.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient: _finish_hunting 0 2023-11-25T05:03:36.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 1 monclient: found mon.noname-a-legacy 2023-11-25T05:03:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.7:6789/0 2023-11-25T05:03:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b400b400 con 0x7f62b400ae70 2023-11-25T05:03:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient: _finish_auth 0 2023-11-25T05:03:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient: _check_auth_tickets 2023-11-25T05:03:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4243585761 0 0) 0x7f62bc011040 con 0x7f62b400ae70 2023-11-25T05:03:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62ce6f2640 5 monclient: authenticate success, global_id 4101 2023-11-25T05:03:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62ce6f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:03:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-11-25T05:03:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient: dump: 2023-11-25T05:03:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:03:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:36.791 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:03:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] but i'm connected to v1:172.21.15.7:6789/0, reconnecting 2023-11-25T05:03:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 >> v1:172.21.15.7:6789/0 conn(0x7f62b400ae70 legacy=0x7f62b4017690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:36.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-25T05:03:36.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62b0003b00 0x7f62b0003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:36.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient(hunting): picked mon.b con 0x7f62b0003b00 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:36.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b00047c0 0x7f62b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:36.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient(hunting): picked mon.c con 0x7f62b00047c0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:36.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62b00052a0 0x7f62b000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:36.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient(hunting): picked mon.a con 0x7f62b00052a0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:36.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:36.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c81421b0 con 0x7f62b00052a0 2023-11-25T05:03:36.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:36.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b000c150 con 0x7f62b0003b00 2023-11-25T05:03:36.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:36.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b000c310 con 0x7f62b00047c0 2023-11-25T05:03:36.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:36.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62c6ffd640 10 monclient(hunting): _finish_auth 0 2023-11-25T05:03:36.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.787+0000 7f62ce6f2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-11-25T05:03:36.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.854+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b00047c0 0x7f62b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:36.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.854+0000 7f62c77fe640 10 monclient(hunting): get_auth_request con 0x7f62b00047c0 auth_method 0 2023-11-25T05:03:36.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.854+0000 7f62c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:36.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.854+0000 7f62c77fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:36.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.854+0000 7f62c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:36.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.855+0000 7f62c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:36.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.855+0000 7f62c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:36.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.855+0000 7f62c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-11-25T05:03:36.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.857+0000 7f62c77fe640 10 monclient(hunting): handle_auth_done global_id 4101 payload 293 2023-11-25T05:03:36.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.857+0000 7f62c77fe640 1 -- 172.21.15.7:0/2834946697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62b0003b00 msgr2=0x7f62b0003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:36.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.857+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62b0003b00 0x7f62b0003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:36.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.857+0000 7f62c77fe640 1 -- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62b00052a0 msgr2=0x7f62b000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:36.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.857+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62b00052a0 0x7f62b000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.857+0000 7f62c77fe640 10 monclient: _finish_hunting 0 2023-11-25T05:03:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.857+0000 7f62c77fe640 1 monclient: found mon.c 2023-11-25T05:03:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.857+0000 7f62c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.857+0000 7f62c77fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:03:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.857+0000 7f62c77fe640 1 -- 172.21.15.7:0/2834946697 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b0003520 con 0x7f62b00047c0 2023-11-25T05:03:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.857+0000 7f62c77fe640 10 monclient: _finish_auth 0 2023-11-25T05:03:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.857+0000 7f62c77fe640 10 monclient: _check_auth_tickets 2023-11-25T05:03:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.857+0000 7f62c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.857+0000 7f62ccc8f640 1 -- 172.21.15.7:0/2834946697 reap_dead start 2023-11-25T05:03:36.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.860+0000 7f62c77fe640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b00047c0 0x7f62b0004c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f62bc000c00 tx=0x7f62b00057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd1ba74c709de47e server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:36.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.879+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc02e620 con 0x7f62b00047c0 2023-11-25T05:03:36.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.879+0000 7f62c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:36.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.879+0000 7f62c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:03:36.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.879+0000 7f62c6ffd640 10 monclient: dump: 2023-11-25T05:03:36.881 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:03:36.881 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:36.881 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:36.881 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:36.881 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:36.881 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:36.881 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:36.881 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:36.881 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:36.881 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:03:36.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.879+0000 7f62ce6f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:03:36.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.924+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62bc02e780 con 0x7f62b00047c0 2023-11-25T05:03:36.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.924+0000 7f62c6ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:36.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.924+0000 7f62c6ffd640 1 -- 172.21.15.7:0/2834946697 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc02eaf0 con 0x7f62b00047c0 2023-11-25T05:03:36.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.924+0000 7f62c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:36.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.924+0000 7f62c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:03:36.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.924+0000 7f62c6ffd640 10 monclient: dump: 2023-11-25T05:03:36.931 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:03:36.932 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:36.932 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:36.932 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:36.932 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:36.932 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:36.932 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:36.932 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:36.932 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:36.932 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:03:36.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.924+0000 7f62ce6f2640 10 monclient: get_monmap_and_config success 2023-11-25T05:03:36.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.924+0000 7f62ce6f2640 4 set_mon_vals no callback set 2023-11-25T05:03:36.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.925+0000 7f62ce6f2640 10 monclient: shutdown 2023-11-25T05:03:36.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.925+0000 7f62ce6f2640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:36.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.925+0000 7f62ce6f2640 1 -- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b00047c0 msgr2=0x7f62b0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:36.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.925+0000 7f62ce6f2640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b00047c0 0x7f62b0004c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f62bc000c00 tx=0x7f62b00057f0 comp rx=0 tx=0).stop 2023-11-25T05:03:36.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.925+0000 7f62ce6f2640 1 -- 172.21.15.7:0/2834946697 shutdown_connections 2023-11-25T05:03:36.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.925+0000 7f62ce6f2640 1 --2- 172.21.15.7:0/2834946697 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b00047c0 0x7f62b0004c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:36.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.925+0000 7f62ce6f2640 1 -- 172.21.15.7:0/2834946697 >> 172.21.15.7:0/2834946697 conn(0x7f62c8061a90 msgr2=0x7f62c80b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:36.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.925+0000 7f62ce6f2640 1 -- 172.21.15.7:0/2834946697 shutdown_connections 2023-11-25T05:03:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.925+0000 7f62ce6f2640 1 -- 172.21.15.7:0/2834946697 wait complete. 2023-11-25T05:03:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.926+0000 7f62ce6f2640 10 monclient: build_initial_monmap 2023-11-25T05:03:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.926+0000 7f62ce6f2640 10 monclient: monmap: 2023-11-25T05:03:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:03:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:03:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:03:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:03:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:03:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:03:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:03:36.935 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:03:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.926+0000 7f62ce6f2640 1 librados: starting msgr at 2023-11-25T05:03:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.926+0000 7f62ce6f2640 1 librados: starting objecter 2023-11-25T05:03:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 1 Processor -- start 2023-11-25T05:03:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 1 -- start start 2023-11-25T05:03:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 1 librados: setting wanted keys 2023-11-25T05:03:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 1 librados: calling monclient init 2023-11-25T05:03:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 10 monclient: init 2023-11-25T05:03:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-25T05:03:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b00047c0 0x7f62c80b5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 10 monclient(hunting): picked mon.noname-c con 0x7f62b00047c0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62c80b5950 0x7f62c81d43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 10 monclient(hunting): picked mon.noname-a con 0x7f62c80b5950 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62c81db120 0x7f62c80b7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 10 monclient(hunting): picked mon.noname-b con 0x7f62c81db120 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c81daa10 con 0x7f62c80b5950 2023-11-25T05:03:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c80b6620 con 0x7f62c81db120 2023-11-25T05:03:36.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:36.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c813bb40 con 0x7f62b00047c0 2023-11-25T05:03:36.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:36.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.927+0000 7f62ce6f2640 10 monclient(hunting): authenticate will time out at 1343.170799s 2023-11-25T05:03:36.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.929+0000 7f62ccc8f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62c81db120 0x7f62c80b7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:36.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.934+0000 7f62c7fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b00047c0 0x7f62c80b5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:36.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.942+0000 7f62ccc8f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62c81db120 0x7f62c80b7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:40732/0 (socket says 172.21.15.7:40732) 2023-11-25T05:03:36.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.942+0000 7f62ccc8f640 1 -- 172.21.15.7:0/2320939548 learned_addr learned my addr 172.21.15.7:0/2320939548 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:03:36.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.942+0000 7f62ccc8f640 10 monclient(hunting): get_auth_request con 0x7f62c81db120 auth_method 0 2023-11-25T05:03:36.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.942+0000 7f62ccc8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:36.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.942+0000 7f62ccc8f640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:36.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.942+0000 7f62ccc8f640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:36.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.944+0000 7f62ccc8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:36.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.944+0000 7f62ccc8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:36.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.944+0000 7f62ccc8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:36.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.947+0000 7f62ccc8f640 10 monclient(hunting): handle_auth_done global_id 4105 payload 824 2023-11-25T05:03:36.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.947+0000 7f62ccc8f640 1 -- 172.21.15.7:0/2320939548 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b00047c0 msgr2=0x7f62c80b5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:36.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.947+0000 7f62ccc8f640 1 --2- 172.21.15.7:0/2320939548 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b00047c0 0x7f62c80b5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:36.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.947+0000 7f62ccc8f640 1 -- 172.21.15.7:0/2320939548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62c80b5950 msgr2=0x7f62c81d43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:36.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.947+0000 7f62ccc8f640 1 --2- 172.21.15.7:0/2320939548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62c80b5950 0x7f62c81d43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:36.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.947+0000 7f62ccc8f640 10 monclient: _finish_hunting 0 2023-11-25T05:03:36.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.947+0000 7f62ccc8f640 1 monclient: found mon.noname-b 2023-11-25T05:03:36.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.947+0000 7f62ccc8f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:36.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.947+0000 7f62ccc8f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:03:36.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:36.947+0000 7f62ccc8f640 1 -- 172.21.15.7:0/2320939548 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c81dac00 con 0x7f62c81db120 2023-11-25T05:03:37.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.011+0000 7f62ccc8f640 1 --2- 172.21.15.7:0/2320939548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62c81db120 0x7f62c80b7ba0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f62b4006180 tx=0x7f62c0013a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:37.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.011+0000 7f62c4ff9640 10 client.?.objecter ms_handle_connect 0x7f62c81db120 2023-11-25T05:03:37.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.011+0000 7f62c4ff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:03:37.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62c4ff9640 1 -- 172.21.15.7:0/2320939548 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c00158a0 con 0x7f62c81db120 2023-11-25T05:03:37.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:37.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62c4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:03:37.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62c4ff9640 10 monclient: dump: 2023-11-25T05:03:37.039 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:03:37.039 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:37.039 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:37.039 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:37.039 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:37.039 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:37.040 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:37.040 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:37.040 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:37.040 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:03:37.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62c4ff9640 10 monclient: _finish_auth 0 2023-11-25T05:03:37.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62c4ff9640 10 monclient: _check_auth_tickets 2023-11-25T05:03:37.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:37.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62c4ff9640 1 -- 172.21.15.7:0/2320939548 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c0015a00 con 0x7f62c81db120 2023-11-25T05:03:37.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62ce6f2640 5 monclient: authenticate success, global_id 4105 2023-11-25T05:03:37.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62ce6f2640 10 monclient: _renew_subs 2023-11-25T05:03:37.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62ce6f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:37.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62ce6f2640 1 -- 172.21.15.7:0/2320939548 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c813c710 con 0x7f62c81db120 2023-11-25T05:03:37.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62c4ff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:37.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62c4ff9640 1 -- 172.21.15.7:0/2320939548 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0015db0 con 0x7f62c81db120 2023-11-25T05:03:37.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:37.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62c4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:03:37.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62c4ff9640 10 monclient: dump: 2023-11-25T05:03:37.041 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:03:37.041 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:37.041 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:37.041 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:37.041 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:37.042 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:37.042 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:37.042 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:37.042 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:37.042 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:03:37.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62c67fc640 4 set_mon_vals no callback set 2023-11-25T05:03:37.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62ce6f2640 10 client.4105.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:03:37.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62ce6f2640 10 monclient: _renew_subs 2023-11-25T05:03:37.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62ce6f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:37.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62ce6f2640 1 -- 172.21.15.7:0/2320939548 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c81d4ad0 con 0x7f62c81db120 2023-11-25T05:03:37.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.036+0000 7f62ce6f2640 1 librados: init done 2023-11-25T05:03:37.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.037+0000 7f62ce6f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:37.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.037+0000 7f62ce6f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:37.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.037+0000 7f62ce6f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:37.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.037+0000 7f62ce6f2640 1 -- 172.21.15.7:0/2320939548 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c8067ac0 con 0x7f62c81db120 2023-11-25T05:03:37.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.209+0000 7f62c4ff9640 1 -- 172.21.15.7:0/2320939548 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f62c0016770 con 0x7f62c81db120 2023-11-25T05:03:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.220+0000 7f62c4ff9640 1 -- 172.21.15.7:0/2320939548 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f62c0020850 con 0x7f62c81db120 2023-11-25T05:03:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.220+0000 7f62c4ff9640 10 client.4105.objecter ms_dispatch 0x7f62c8000b90 osd_map(1..1 src has 1..1) v4 2023-11-25T05:03:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.220+0000 7f62c4ff9640 3 client.4105.objecter handle_osd_map got epochs [1,1] > 0 2023-11-25T05:03:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.220+0000 7f62c4ff9640 3 client.4105.objecter handle_osd_map decoding full epoch 1 2023-11-25T05:03:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.220+0000 7f62c4ff9640 20 client.4105.objecter dump_active .. 0 homeless 2023-11-25T05:03:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.220+0000 7f62c4ff9640 1 -- 172.21.15.7:0/2320939548 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f62c000e040 con 0x7f62c81db120 2023-11-25T05:03:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.220+0000 7f62c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.220+0000 7f62c4ff9640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:03:37.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.289+0000 7f62ce6f2640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-11-25T05:03:37.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.289+0000 7f62ce6f2640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-11-25T05:03:37.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.289+0000 7f62ce6f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:37.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.289+0000 7f62ce6f2640 1 -- 172.21.15.7:0/2320939548 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f62c8075b60 con 0x7f62c81db120 2023-11-25T05:03:37.423 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:37.420+0000 7fdccda30040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-25T05:03:37.485 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:37.483+0000 7fdccda30040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-25T05:03:37.488 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:37.487+0000 7f23a2432040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-25T05:03:37.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.542+0000 7f62c4ff9640 1 -- 172.21.15.7:0/2320939548 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f62c0016bb0 con 0x7f62c81db120 2023-11-25T05:03:37.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.542+0000 7f62c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-11-25T05:03:37.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.542+0000 7f62c4ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-11-25T05:03:37.544 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2023-11-25T05:03:37.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 10 librados: watch_flush enter 2023-11-25T05:03:37.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 10 librados: watch_flush exit 2023-11-25T05:03:37.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 20 client.4105.objecter shutdown clearing up homeless session... 2023-11-25T05:03:37.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 10 client.4105.objecter successfully canceled tick 2023-11-25T05:03:37.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 10 monclient: shutdown 2023-11-25T05:03:37.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:37.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 1 -- 172.21.15.7:0/2320939548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62c81db120 msgr2=0x7f62c80b7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:37.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 1 --2- 172.21.15.7:0/2320939548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62c81db120 0x7f62c80b7ba0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f62b4006180 tx=0x7f62c0013a40 comp rx=0 tx=0).stop 2023-11-25T05:03:37.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 1 -- 172.21.15.7:0/2320939548 shutdown_connections 2023-11-25T05:03:37.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 1 --2- 172.21.15.7:0/2320939548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62c81db120 0x7f62c80b7ba0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:37.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 1 --2- 172.21.15.7:0/2320939548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62c80b5950 0x7f62c81d43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:37.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 1 --2- 172.21.15.7:0/2320939548 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f62b00047c0 0x7f62c80b5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:37.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 1 -- 172.21.15.7:0/2320939548 >> 172.21.15.7:0/2320939548 conn(0x7f62c8061a90 msgr2=0x7f62c8073c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:37.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 1 -- 172.21.15.7:0/2320939548 shutdown_connections 2023-11-25T05:03:37.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 1 -- 172.21.15.7:0/2320939548 wait complete. 2023-11-25T05:03:37.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:37.543+0000 7f62ce6f2640 1 librados: shutdown 2023-11-25T05:03:37.550 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:37.548+0000 7f23a2432040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-25T05:03:37.556 INFO:tasks.ceph:check_enable_crimson: False 2023-11-25T05:03:37.556 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-25T05:03:37.557 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:03:37.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-25T05:03:37.591 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:03:37.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-25T05:03:37.663 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:03:37.663 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-25T05:03:37.737 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:03:37.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-11-25T05:03:37.808 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:03:37.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-11-25T05:03:37.835 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:03:37.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-11-25T05:03:37.902 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:03:37.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-11-25T05:03:37.970 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-25T05:03:37.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-11-25T05:03:38.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 5c87807e-7b82-4866-ba36-60b44a25f317 0 2023-11-25T05:03:38.077 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:38.076+0000 7fdccda30040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-25T05:03:38.130 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:38.129+0000 7f23a2432040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-25T05:03:38.132 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:38.131+0000 7fdccda30040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-25T05:03:38.176 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.174+0000 7f393b42d640 10 monclient: get_monmap_and_config 2023-11-25T05:03:38.176 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.174+0000 7f393b42d640 10 monclient: build_initial_monmap 2023-11-25T05:03:38.176 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.174+0000 7f393b42d640 10 monclient: monmap: 2023-11-25T05:03:38.176 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:38.176 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:38.176 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:03:38.175895+0000 2023-11-25T05:03:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:03:38.175895+0000 2023-11-25T05:03:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:03:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:03:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:03:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:03:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:03:38.177 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:38.179 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.175+0000 7f393b42d640 1 Processor -- start 2023-11-25T05:03:38.179 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.175+0000 7f393b42d640 1 -- start start 2023-11-25T05:03:38.179 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.175+0000 7f393b42d640 10 monclient: init 2023-11-25T05:03:38.179 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.176+0000 7f393b42d640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:38.179 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.176+0000 7f393b42d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-11-25T05:03:38.179 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.176+0000 7f393b42d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f39340a8a00 0x7f39340a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.176+0000 7f393b42d640 10 monclient(hunting): picked mon.noname-a con 0x7f39340a8a00 addr v2:172.21.15.7:3300/0 2023-11-25T05:03:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.176+0000 7f393b42d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39340b2260 addr v1:172.21.15.7:6789/0 2023-11-25T05:03:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.176+0000 7f393b42d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f393413f080 0x7f393413f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.176+0000 7f393b42d640 10 monclient(hunting): picked mon.noname-b con 0x7f393413f080 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.176+0000 7f393b42d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39340aa970 con 0x7f39340b2260 2023-11-25T05:03:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.176+0000 7f393b42d640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.176+0000 7f393b42d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39341421b0 con 0x7f39340a8a00 2023-11-25T05:03:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.176+0000 7f393b42d640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.176+0000 7f393b42d640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39340aac50 con 0x7f393413f080 2023-11-25T05:03:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.176+0000 7f393b42d640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.176+0000 7f393b42d640 10 monclient(hunting): authenticate will time out at 1319.542872s 2023-11-25T05:03:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.177+0000 7f39399ca640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f393413f080 0x7f393413f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.177+0000 7f39391c9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f39340a8a00 0x7f39340a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.177+0000 7f39399ca640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f393413f080 0x7f393413f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.90:60828/0 (socket says 172.21.15.90:60828) 2023-11-25T05:03:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.177+0000 7f39399ca640 1 -- 172.21.15.90:0/1820883936 learned_addr learned my addr 172.21.15.90:0/1820883936 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-11-25T05:03:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.177+0000 7f39399ca640 10 monclient(hunting): get_auth_request con 0x7f393413f080 auth_method 0 2023-11-25T05:03:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.177+0000 7f39399ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.177+0000 7f39399ca640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.177+0000 7f39399ca640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.177+0000 7f39391c9640 10 monclient(hunting): get_auth_request con 0x7f39340a8a00 auth_method 0 2023-11-25T05:03:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.177+0000 7f39391c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.177+0000 7f39391c9640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.177+0000 7f39391c9640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:38.181 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:38.178+0000 7fdccda30040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-25T05:03:38.182 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.178+0000 7f39399ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:38.182 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.178+0000 7f39399ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:38.183 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.178+0000 7f39399ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:38.183 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.179+0000 7f39391c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:38.183 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.179+0000 7f39391c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:38.183 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.179+0000 7f39391c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:38.183 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.180+0000 7f39399ca640 10 monclient(hunting): handle_auth_done global_id 4103 payload 274 2023-11-25T05:03:38.183 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.180+0000 7f39399ca640 1 -- 172.21.15.90:0/1820883936 >> v2:172.21.15.7:3300/0 conn(0x7f39340a8a00 msgr2=0x7f39340a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:38.183 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.180+0000 7f39399ca640 1 --2- 172.21.15.90:0/1820883936 >> v2:172.21.15.7:3300/0 conn(0x7f39340a8a00 0x7f39340a8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.183 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.180+0000 7f39399ca640 1 -- 172.21.15.90:0/1820883936 >> v1:172.21.15.7:6789/0 conn(0x7f39340b2260 legacy=0x7f39341425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:38.183 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.180+0000 7f39399ca640 10 monclient: _finish_hunting 0 2023-11-25T05:03:38.183 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.180+0000 7f39399ca640 1 monclient: found mon.noname-b 2023-11-25T05:03:38.183 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.180+0000 7f39399ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:38.183 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.180+0000 7f39399ca640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:03:38.183 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.180+0000 7f39399ca640 1 -- 172.21.15.90:0/1820883936 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39340aaf30 con 0x7f393413f080 2023-11-25T05:03:38.184 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.183+0000 7f39399ca640 1 --2- 172.21.15.90:0/1820883936 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f393413f080 0x7f393413f530 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f392c009e50 tx=0x7f392c02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d6461425cc98486 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:38.186 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:38.185+0000 7f23a2432040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-25T05:03:38.190 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f392bfff640 1 -- 172.21.15.90:0/1820883936 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c02c9e0 con 0x7f393413f080 2023-11-25T05:03:38.190 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f392bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:38.190 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f392bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:03:38.190 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f392bfff640 10 monclient: dump: 2023-11-25T05:03:38.190 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:38.190 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:38.190 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.190 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.190 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:38.191 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f392bfff640 10 monclient: _finish_auth 0 2023-11-25T05:03:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f392bfff640 10 monclient: _check_auth_tickets 2023-11-25T05:03:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f392bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f392bfff640 1 -- 172.21.15.90:0/1820883936 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392c02cb40 con 0x7f393413f080 2023-11-25T05:03:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f392bfff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f392bfff640 1 -- 172.21.15.90:0/1820883936 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c0343f0 con 0x7f393413f080 2023-11-25T05:03:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f393b42d640 5 monclient: authenticate success, global_id 4103 2023-11-25T05:03:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f393b42d640 10 monclient: get_monmap_and_config success 2023-11-25T05:03:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f393b42d640 4 set_mon_vals no callback set 2023-11-25T05:03:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f392bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f392bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:03:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.188+0000 7f392bfff640 10 monclient: dump: 2023-11-25T05:03:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:38.193 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:38.196 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.189+0000 7f393b42d640 10 monclient: shutdown 2023-11-25T05:03:38.196 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.189+0000 7f393b42d640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:38.196 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.189+0000 7f393b42d640 1 -- 172.21.15.90:0/1820883936 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f393413f080 msgr2=0x7f393413f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:38.196 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.189+0000 7f393b42d640 1 --2- 172.21.15.90:0/1820883936 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f393413f080 0x7f393413f530 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f392c009e50 tx=0x7f392c02be40 comp rx=0 tx=0).stop 2023-11-25T05:03:38.197 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.189+0000 7f393b42d640 1 -- 172.21.15.90:0/1820883936 shutdown_connections 2023-11-25T05:03:38.197 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.189+0000 7f393b42d640 1 --2- 172.21.15.90:0/1820883936 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f393413f080 0x7f393413f530 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.197 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.189+0000 7f393b42d640 1 --2- 172.21.15.90:0/1820883936 >> v2:172.21.15.7:3300/0 conn(0x7f39340a8a00 0x7f39340a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.197 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.189+0000 7f393b42d640 1 -- 172.21.15.90:0/1820883936 >> 172.21.15.90:0/1820883936 conn(0x7f3934061a90 msgr2=0x7f39340b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:38.197 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.189+0000 7f393b42d640 1 -- 172.21.15.90:0/1820883936 shutdown_connections 2023-11-25T05:03:38.197 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.189+0000 7f393b42d640 1 -- 172.21.15.90:0/1820883936 wait complete. 2023-11-25T05:03:38.197 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.190+0000 7f393b42d640 10 monclient: build_initial_monmap 2023-11-25T05:03:38.197 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.190+0000 7f393b42d640 10 monclient: monmap: 2023-11-25T05:03:38.197 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:38.197 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:03:38.197 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2023-11-25T05:03:38.198 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2023-11-25T05:03:38.198 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:38.198 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:38.198 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:03:38.198 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:03:38.198 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:03:38.198 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:38.198 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.190+0000 7f393b42d640 1 librados: starting msgr at 2023-11-25T05:03:38.198 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.190+0000 7f393b42d640 1 librados: starting objecter 2023-11-25T05:03:38.198 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.190+0000 7f393b42d640 1 Processor -- start 2023-11-25T05:03:38.198 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.190+0000 7f393b42d640 1 -- start start 2023-11-25T05:03:38.198 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.190+0000 7f393b42d640 1 librados: setting wanted keys 2023-11-25T05:03:38.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.190+0000 7f393b42d640 1 librados: calling monclient init 2023-11-25T05:03:38.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.190+0000 7f393b42d640 10 monclient: init 2023-11-25T05:03:38.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:38.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-25T05:03:38.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39340a8a00 0x7f393413a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:38.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 10 monclient(hunting): picked mon.noname-c con 0x7f39340a8a00 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:38.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39340b2260 0x7f393413ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:38.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 10 monclient(hunting): picked mon.noname-a con 0x7f39340b2260 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:38.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f393413f080 0x7f393413bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:38.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 10 monclient(hunting): picked mon.noname-b con 0x7f393413f080 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:38.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:38.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3934145cb0 con 0x7f39340b2260 2023-11-25T05:03:38.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:38.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3934144ab0 con 0x7f393413f080 2023-11-25T05:03:38.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:38.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39341460d0 con 0x7f39340a8a00 2023-11-25T05:03:38.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:38.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f393b42d640 10 monclient(hunting): authenticate will time out at 1319.557643s 2023-11-25T05:03:38.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f39399ca640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f393413f080 0x7f393413bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:38.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f39389c8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39340b2260 0x7f393413ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:38.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.191+0000 7f39391c9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39340a8a00 0x7f393413a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:38.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39391c9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39340a8a00 0x7f393413a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.90:60832/0 (socket says 172.21.15.90:60832) 2023-11-25T05:03:38.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39391c9640 1 -- 172.21.15.90:0/3355584415 learned_addr learned my addr 172.21.15.90:0/3355584415 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-11-25T05:03:38.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39391c9640 10 monclient(hunting): get_auth_request con 0x7f39340a8a00 auth_method 0 2023-11-25T05:03:38.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39391c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:38.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39391c9640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:38.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39391c9640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:38.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39389c8640 10 monclient(hunting): get_auth_request con 0x7f39340b2260 auth_method 0 2023-11-25T05:03:38.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39389c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:38.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39389c8640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:38.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39389c8640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:38.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39399ca640 10 monclient(hunting): get_auth_request con 0x7f393413f080 auth_method 0 2023-11-25T05:03:38.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39399ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:38.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39399ca640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:38.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39399ca640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:38.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39391c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:38.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39391c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:38.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39391c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:38.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39389c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:38.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39389c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:38.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.192+0000 7f39389c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:38.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.193+0000 7f39399ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:38.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.193+0000 7f39399ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:38.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.193+0000 7f39399ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.196+0000 7f39389c8640 10 monclient(hunting): handle_auth_done global_id 4119 payload 824 2023-11-25T05:03:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.196+0000 7f39389c8640 1 -- 172.21.15.90:0/3355584415 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39340a8a00 msgr2=0x7f393413a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.196+0000 7f39389c8640 1 --2- 172.21.15.90:0/3355584415 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39340a8a00 0x7f393413a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.196+0000 7f39389c8640 1 -- 172.21.15.90:0/3355584415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f393413f080 msgr2=0x7f393413bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.196+0000 7f39389c8640 1 --2- 172.21.15.90:0/3355584415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f393413f080 0x7f393413bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.196+0000 7f39389c8640 10 monclient: _finish_hunting 0 2023-11-25T05:03:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.196+0000 7f39389c8640 1 monclient: found mon.noname-a 2023-11-25T05:03:38.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.196+0000 7f39389c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:38.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.197+0000 7f39389c8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:03:38.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.197+0000 7f39389c8640 1 -- 172.21.15.90:0/3355584415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c009b30 con 0x7f39340b2260 2023-11-25T05:03:38.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.197+0000 7f39391c9640 1 --2- 172.21.15.90:0/3355584415 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39340a8a00 0x7f393413a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-25T05:03:38.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.200+0000 7f39389c8640 1 --2- 172.21.15.90:0/3355584415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39340b2260 0x7f393413ba70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f393000c340 tx=0x7f393000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:38.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.200+0000 7f3929ffb640 10 client.?.objecter ms_handle_connect 0x7f39340b2260 2023-11-25T05:03:38.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.200+0000 7f3929ffb640 10 client.?.objecter resend_mon_ops 2023-11-25T05:03:38.206 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f3929ffb640 1 -- 172.21.15.90:0/3355584415 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39300052d0 con 0x7f39340b2260 2023-11-25T05:03:38.206 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f3929ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:38.206 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f3929ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:03:38.206 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f3929ffb640 10 monclient: dump: 2023-11-25T05:03:38.206 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:38.206 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:38.207 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.207 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.207 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:38.207 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:38.207 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:38.207 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:38.207 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:38.209 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f3929ffb640 10 monclient: _finish_auth 0 2023-11-25T05:03:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f3929ffb640 10 monclient: _check_auth_tickets 2023-11-25T05:03:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f3929ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f3929ffb640 1 -- 172.21.15.90:0/3355584415 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3930005450 con 0x7f39340b2260 2023-11-25T05:03:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f393b42d640 5 monclient: authenticate success, global_id 4119 2023-11-25T05:03:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f393b42d640 10 monclient: _renew_subs 2023-11-25T05:03:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f393b42d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:03:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f393b42d640 1 -- 172.21.15.90:0/3355584415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393413c4f0 con 0x7f39340b2260 2023-11-25T05:03:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f3929ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f3929ffb640 1 -- 172.21.15.90:0/3355584415 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930007260 con 0x7f39340b2260 2023-11-25T05:03:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f3929ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f3929ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:03:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f3929ffb640 10 monclient: dump: 2023-11-25T05:03:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:38.210 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f392b7fe640 4 set_mon_vals no callback set 2023-11-25T05:03:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f393b42d640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:03:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f393b42d640 10 monclient: _renew_subs 2023-11-25T05:03:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f393b42d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:03:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f393b42d640 1 -- 172.21.15.90:0/3355584415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3934139720 con 0x7f39340b2260 2023-11-25T05:03:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.204+0000 7f393b42d640 1 librados: init done 2023-11-25T05:03:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.205+0000 7f393b42d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.205+0000 7f393b42d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.205+0000 7f393b42d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:03:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.205+0000 7f393b42d640 1 -- 172.21.15.90:0/3355584415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3934067ac0 con 0x7f39340b2260 2023-11-25T05:03:38.236 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:38.234+0000 7f23a2432040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-25T05:03:38.238 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:38.237+0000 7fdccda30040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-25T05:03:38.280 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:38.279+0000 7fdccda30040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-25T05:03:38.293 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:38.292+0000 7f23a2432040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-25T05:03:38.327 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.325+0000 7f3929ffb640 1 -- 172.21.15.90:0/3355584415 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f39300067d0 con 0x7f39340b2260 2023-11-25T05:03:38.333 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.332+0000 7f3929ffb640 1 -- 172.21.15.90:0/3355584415 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3930003940 con 0x7f39340b2260 2023-11-25T05:03:38.333 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.332+0000 7f3929ffb640 10 client.4119.objecter ms_dispatch 0x7f3934000b90 osd_map(2..2 src has 1..2) v4 2023-11-25T05:03:38.334 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.332+0000 7f3929ffb640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2023-11-25T05:03:38.334 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.332+0000 7f3929ffb640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2023-11-25T05:03:38.334 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.332+0000 7f3929ffb640 20 client.4119.objecter dump_active .. 0 homeless 2023-11-25T05:03:38.334 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.332+0000 7f3929ffb640 1 -- 172.21.15.90:0/3355584415 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f3930020060 con 0x7f39340b2260 2023-11-25T05:03:38.334 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.332+0000 7f3929ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:38.334 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.332+0000 7f3929ffb640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:03:38.339 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:38.338+0000 7f23a2432040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-25T05:03:38.384 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:38.382+0000 7fdccda30040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-25T05:03:38.406 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.404+0000 7f393b42d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5c87807e-7b82-4866-ba36-60b44a25f317", "id": 0}] 2023-11-25T05:03:38.406 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.404+0000 7f393b42d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5c87807e-7b82-4866-ba36-60b44a25f317", "id": 0}] 2023-11-25T05:03:38.406 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.404+0000 7f393b42d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:03:38.406 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.404+0000 7f393b42d640 1 -- 172.21.15.90:0/3355584415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5c87807e-7b82-4866-ba36-60b44a25f317", "id": 0} v 0) v1 -- 0x7f3934075b60 con 0x7f39340b2260 2023-11-25T05:03:38.447 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:38.445+0000 7f23a2432040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-25T05:03:38.453 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:38.452+0000 7fdccda30040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-25T05:03:38.497 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.495+0000 7f3929ffb640 1 -- 172.21.15.90:0/3355584415 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5c87807e-7b82-4866-ba36-60b44a25f317", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3930006bc0 con 0x7f39340b2260 2023-11-25T05:03:38.497 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.496+0000 7f3929ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5c87807e-7b82-4866-ba36-60b44a25f317", "id": 0}] 2023-11-25T05:03:38.497 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.496+0000 7f3929ffb640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:03:38.498 INFO:teuthology.orchestra.run.smithi090.stdout:0 2023-11-25T05:03:38.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.496+0000 7f393b42d640 10 librados: watch_flush enter 2023-11-25T05:03:38.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.496+0000 7f393b42d640 10 librados: watch_flush exit 2023-11-25T05:03:38.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.496+0000 7f393b42d640 20 client.4119.objecter shutdown clearing up homeless session... 2023-11-25T05:03:38.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.496+0000 7f393b42d640 10 client.4119.objecter successfully canceled tick 2023-11-25T05:03:38.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.496+0000 7f393b42d640 10 monclient: shutdown 2023-11-25T05:03:38.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.496+0000 7f393b42d640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:38.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.496+0000 7f393b42d640 1 -- 172.21.15.90:0/3355584415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39340b2260 msgr2=0x7f393413ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:38.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.497+0000 7f393b42d640 1 --2- 172.21.15.90:0/3355584415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39340b2260 0x7f393413ba70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f393000c340 tx=0x7f393000c800 comp rx=0 tx=0).stop 2023-11-25T05:03:38.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.497+0000 7f393b42d640 1 -- 172.21.15.90:0/3355584415 shutdown_connections 2023-11-25T05:03:38.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.497+0000 7f393b42d640 1 --2- 172.21.15.90:0/3355584415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f393413f080 0x7f393413bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.497+0000 7f393b42d640 1 --2- 172.21.15.90:0/3355584415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39340b2260 0x7f393413ba70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.497+0000 7f393b42d640 1 --2- 172.21.15.90:0/3355584415 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39340a8a00 0x7f393413a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.497+0000 7f393b42d640 1 -- 172.21.15.90:0/3355584415 >> 172.21.15.90:0/3355584415 conn(0x7f3934061a90 msgr2=0x7f3934134720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:38.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.497+0000 7f393b42d640 1 -- 172.21.15.90:0/3355584415 shutdown_connections 2023-11-25T05:03:38.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.497+0000 7f393b42d640 1 -- 172.21.15.90:0/3355584415 wait complete. 2023-11-25T05:03:38.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.497+0000 7f393b42d640 1 librados: shutdown 2023-11-25T05:03:38.500 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:38.499+0000 7fdccda30040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-25T05:03:38.508 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 0e252ab8-9524-4ccc-b08f-d95bbf05fbea 1 2023-11-25T05:03:38.523 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:38.522+0000 7f23a2432040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-25T05:03:38.557 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:38.555+0000 7fdccda30040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-25T05:03:38.575 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:38.574+0000 7f23a2432040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-25T05:03:38.607 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:38.605+0000 7fdccda30040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-25T05:03:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.606+0000 7fa325da0640 10 monclient: get_monmap_and_config 2023-11-25T05:03:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.606+0000 7fa325da0640 10 monclient: build_initial_monmap 2023-11-25T05:03:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.606+0000 7fa325da0640 10 monclient: monmap: 2023-11-25T05:03:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:03:38.607555+0000 2023-11-25T05:03:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:03:38.607555+0000 2023-11-25T05:03:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:03:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:03:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:03:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:03:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:03:38.609 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:38.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 1 Processor -- start 2023-11-25T05:03:38.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 1 -- start start 2023-11-25T05:03:38.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 10 monclient: init 2023-11-25T05:03:38.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:38.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-11-25T05:03:38.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa32006fd50 0x7fa320071e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:38.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 10 monclient(hunting): picked mon.noname-c con 0x7fa32006fd50 addr v2:172.21.15.90:3300/0 2023-11-25T05:03:38.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3200ae820 addr v1:172.21.15.90:6789/0 2023-11-25T05:03:38.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3200b0170 0x7fa3200b0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:38.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 10 monclient(hunting): picked mon.noname-b con 0x7fa3200b0170 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:38.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3200717a0 con 0x7fa3200ae820 2023-11-25T05:03:38.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:38.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa320070130 con 0x7fa32006fd50 2023-11-25T05:03:38.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:38.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3200719f0 con 0x7fa3200b0170 2023-11-25T05:03:38.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:38.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.607+0000 7fa325da0640 10 monclient(hunting): authenticate will time out at 1319.974518s 2023-11-25T05:03:38.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.608+0000 7fa32559f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3200b0170 0x7fa3200b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:38.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.608+0000 7fa324d9e640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa32006fd50 0x7fa320071e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:38.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.608+0000 7fa324d9e640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa32006fd50 0x7fa320071e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40368/0 (socket says 172.21.15.90:40368) 2023-11-25T05:03:38.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.608+0000 7fa324d9e640 1 -- 172.21.15.90:0/3447216224 learned_addr learned my addr 172.21.15.90:0/3447216224 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-11-25T05:03:38.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.608+0000 7fa324d9e640 10 monclient(hunting): get_auth_request con 0x7fa32006fd50 auth_method 0 2023-11-25T05:03:38.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.608+0000 7fa324d9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:38.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.608+0000 7fa324d9e640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:38.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.608+0000 7fa324d9e640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:38.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.609+0000 7fa32559f640 10 monclient(hunting): get_auth_request con 0x7fa3200b0170 auth_method 0 2023-11-25T05:03:38.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.609+0000 7fa32559f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:38.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.609+0000 7fa32559f640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:38.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.609+0000 7fa32559f640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:38.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.610+0000 7fa324d9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:38.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.610+0000 7fa324d9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:38.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.610+0000 7fa324d9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:38.615 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.610+0000 7fa32559f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.610+0000 7fa32559f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.610+0000 7fa32559f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.611+0000 7fa324d9e640 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2023-11-25T05:03:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.611+0000 7fa324d9e640 1 -- 172.21.15.90:0/3447216224 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3200b0170 msgr2=0x7fa3200b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.611+0000 7fa324d9e640 1 --2- 172.21.15.90:0/3447216224 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3200b0170 0x7fa3200b0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.611+0000 7fa324d9e640 1 -- 172.21.15.90:0/3447216224 >> v1:172.21.15.90:6789/0 conn(0x7fa3200ae820 legacy=0x7fa32013eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.611+0000 7fa324d9e640 10 monclient: _finish_hunting 0 2023-11-25T05:03:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.611+0000 7fa324d9e640 1 monclient: found mon.noname-c 2023-11-25T05:03:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.611+0000 7fa324d9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.611+0000 7fa324d9e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:03:38.617 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.611+0000 7fa324d9e640 1 -- 172.21.15.90:0/3447216224 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa320070350 con 0x7fa32006fd50 2023-11-25T05:03:38.618 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.617+0000 7fa324d9e640 1 --2- 172.21.15.90:0/3447216224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa32006fd50 0x7fa320071e10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa31c009870 tx=0x7fa31c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c46edd1e8335e04c server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa3177fe640 1 -- 172.21.15.90:0/3447216224 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31c030840 con 0x7fa32006fd50 2023-11-25T05:03:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa3177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa3177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:03:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa3177fe640 10 monclient: dump: 2023-11-25T05:03:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:38.627 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa3177fe640 10 monclient: _finish_auth 0 2023-11-25T05:03:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa3177fe640 10 monclient: _check_auth_tickets 2023-11-25T05:03:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa3177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa3177fe640 1 -- 172.21.15.90:0/3447216224 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa31c0309a0 con 0x7fa32006fd50 2023-11-25T05:03:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa3177fe640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa3177fe640 1 -- 172.21.15.90:0/3447216224 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31c030d30 con 0x7fa32006fd50 2023-11-25T05:03:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa325da0640 5 monclient: authenticate success, global_id 4111 2023-11-25T05:03:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa325da0640 10 monclient: get_monmap_and_config success 2023-11-25T05:03:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa325da0640 4 set_mon_vals no callback set 2023-11-25T05:03:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa3177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa3177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:03:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.624+0000 7fa3177fe640 10 monclient: dump: 2023-11-25T05:03:38.628 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:38.628 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:38.628 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.628 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.628 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:38.628 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:38.628 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:38.628 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:38.628 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:38.628 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:38.630 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.625+0000 7fa325da0640 10 monclient: shutdown 2023-11-25T05:03:38.631 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.625+0000 7fa325da0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:38.631 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.625+0000 7fa325da0640 1 -- 172.21.15.90:0/3447216224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa32006fd50 msgr2=0x7fa320071e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:38.631 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.625+0000 7fa325da0640 1 --2- 172.21.15.90:0/3447216224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa32006fd50 0x7fa320071e10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa31c009870 tx=0x7fa31c02b980 comp rx=0 tx=0).stop 2023-11-25T05:03:38.631 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.625+0000 7fa325da0640 1 -- 172.21.15.90:0/3447216224 shutdown_connections 2023-11-25T05:03:38.631 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.625+0000 7fa325da0640 1 --2- 172.21.15.90:0/3447216224 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3200b0170 0x7fa3200b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.631 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.625+0000 7fa325da0640 1 --2- 172.21.15.90:0/3447216224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa32006fd50 0x7fa320071e10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.631 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.625+0000 7fa325da0640 1 -- 172.21.15.90:0/3447216224 >> 172.21.15.90:0/3447216224 conn(0x7fa32006bf90 msgr2=0x7fa3200b4660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:38.631 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.625+0000 7fa325da0640 1 -- 172.21.15.90:0/3447216224 shutdown_connections 2023-11-25T05:03:38.631 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.625+0000 7fa325da0640 1 -- 172.21.15.90:0/3447216224 wait complete. 2023-11-25T05:03:38.631 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.625+0000 7fa325da0640 10 monclient: build_initial_monmap 2023-11-25T05:03:38.631 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.625+0000 7fa325da0640 10 monclient: monmap: 2023-11-25T05:03:38.631 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:38.632 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:03:38.632 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2023-11-25T05:03:38.632 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2023-11-25T05:03:38.632 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:38.632 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:38.632 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:03:38.632 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:03:38.632 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:03:38.632 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:38.632 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.626+0000 7fa325da0640 1 librados: starting msgr at 2023-11-25T05:03:38.632 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.626+0000 7fa325da0640 1 librados: starting objecter 2023-11-25T05:03:38.632 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.626+0000 7fa325da0640 1 Processor -- start 2023-11-25T05:03:38.632 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.626+0000 7fa325da0640 1 -- start start 2023-11-25T05:03:38.633 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.626+0000 7fa325da0640 1 librados: setting wanted keys 2023-11-25T05:03:38.633 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.626+0000 7fa325da0640 1 librados: calling monclient init 2023-11-25T05:03:38.633 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.626+0000 7fa325da0640 10 monclient: init 2023-11-25T05:03:38.633 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.626+0000 7fa325da0640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:38.633 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.626+0000 7fa325da0640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:03:38.633 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.627+0000 7fa325da0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa32006fd50 0x7fa320139a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:38.633 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.627+0000 7fa325da0640 10 monclient(hunting): picked mon.noname-c con 0x7fa32006fd50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:38.633 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.627+0000 7fa325da0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3200ae820 0x7fa3200aa130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:38.633 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.627+0000 7fa325da0640 10 monclient(hunting): picked mon.noname-b con 0x7fa3200ae820 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:38.633 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.627+0000 7fa325da0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3200b0170 0x7fa3201d6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:38.633 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.627+0000 7fa325da0640 10 monclient(hunting): picked mon.noname-a con 0x7fa3200b0170 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:38.633 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.627+0000 7fa325da0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:38.633 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.627+0000 7fa325da0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa320140c50 con 0x7fa3200b0170 2023-11-25T05:03:38.634 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.627+0000 7fa325da0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:38.634 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.627+0000 7fa325da0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32013a400 con 0x7fa3200ae820 2023-11-25T05:03:38.634 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.627+0000 7fa325da0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:38.634 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.627+0000 7fa325da0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3200aa780 con 0x7fa32006fd50 2023-11-25T05:03:38.634 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.627+0000 7fa325da0640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:38.634 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.627+0000 7fa325da0640 10 monclient(hunting): authenticate will time out at 1319.993598s 2023-11-25T05:03:38.634 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.628+0000 7fa317fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3200ae820 0x7fa3200aa130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:38.635 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:38.630+0000 7f23a2432040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-25T05:03:38.635 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.630+0000 7fa317fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3200ae820 0x7fa3200aa130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40382/0 (socket says 172.21.15.90:40382) 2023-11-25T05:03:38.636 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.630+0000 7fa317fff640 1 -- 172.21.15.90:0/2266481325 learned_addr learned my addr 172.21.15.90:0/2266481325 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-11-25T05:03:38.636 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.630+0000 7fa317fff640 10 monclient(hunting): get_auth_request con 0x7fa3200ae820 auth_method 0 2023-11-25T05:03:38.636 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.630+0000 7fa317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:38.636 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.630+0000 7fa317fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:38.636 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.630+0000 7fa317fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:38.636 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.630+0000 7fa317fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:38.636 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.630+0000 7fa317fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:38.636 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.630+0000 7fa317fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:38.636 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.633+0000 7fa317fff640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2023-11-25T05:03:38.637 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.633+0000 7fa317fff640 1 -- 172.21.15.90:0/2266481325 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa32006fd50 msgr2=0x7fa320139a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:38.637 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.633+0000 7fa317fff640 1 --2- 172.21.15.90:0/2266481325 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa32006fd50 0x7fa320139a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.637 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.633+0000 7fa317fff640 1 -- 172.21.15.90:0/2266481325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3200b0170 msgr2=0x7fa3201d6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:38.637 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.633+0000 7fa317fff640 1 --2- 172.21.15.90:0/2266481325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3200b0170 0x7fa3201d6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.637 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.633+0000 7fa317fff640 10 monclient: _finish_hunting 0 2023-11-25T05:03:38.637 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.633+0000 7fa317fff640 1 monclient: found mon.noname-b 2023-11-25T05:03:38.637 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.633+0000 7fa317fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:38.637 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.633+0000 7fa317fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:03:38.637 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.633+0000 7fa317fff640 1 -- 172.21.15.90:0/2266481325 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31c0094d0 con 0x7fa3200ae820 2023-11-25T05:03:38.654 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.653+0000 7fa317fff640 1 --2- 172.21.15.90:0/2266481325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3200ae820 0x7fa3200aa130 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa31000c450 tx=0x7fa31000c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:38.654 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.653+0000 7fa3157fa640 10 client.?.objecter ms_handle_connect 0x7fa3200ae820 2023-11-25T05:03:38.654 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.653+0000 7fa3157fa640 10 client.?.objecter resend_mon_ops 2023-11-25T05:03:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.656+0000 7fa3157fa640 1 -- 172.21.15.90:0/2266481325 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3100052d0 con 0x7fa3200ae820 2023-11-25T05:03:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.656+0000 7fa3157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:38.658 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.656+0000 7fa3157fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:03:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.656+0000 7fa3157fa640 10 monclient: dump: 2023-11-25T05:03:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:38.659 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.656+0000 7fa3157fa640 10 monclient: _finish_auth 0 2023-11-25T05:03:38.659 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.656+0000 7fa3157fa640 10 monclient: _check_auth_tickets 2023-11-25T05:03:38.660 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.656+0000 7fa3157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:38.660 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.656+0000 7fa3157fa640 1 -- 172.21.15.90:0/2266481325 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa310005430 con 0x7fa3200ae820 2023-11-25T05:03:38.660 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.656+0000 7fa3157fa640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:38.660 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.656+0000 7fa3157fa640 1 -- 172.21.15.90:0/2266481325 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa310003830 con 0x7fa3200ae820 2023-11-25T05:03:38.660 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.656+0000 7fa3157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:38.660 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa3157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:03:38.660 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa3157fa640 10 monclient: dump: 2023-11-25T05:03:38.660 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:38.660 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:38.660 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.660 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:38.660 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:38.661 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:38.661 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:38.661 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:38.661 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:38.661 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:38.661 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa325da0640 5 monclient: authenticate success, global_id 4114 2023-11-25T05:03:38.661 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa325da0640 10 monclient: _renew_subs 2023-11-25T05:03:38.661 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa325da0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:38.661 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa325da0640 1 -- 172.21.15.90:0/2266481325 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3200aaa60 con 0x7fa3200ae820 2023-11-25T05:03:38.661 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa316ffd640 4 set_mon_vals no callback set 2023-11-25T05:03:38.661 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa325da0640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:03:38.661 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa325da0640 10 monclient: _renew_subs 2023-11-25T05:03:38.661 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa325da0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:38.662 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa325da0640 1 -- 172.21.15.90:0/2266481325 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3200ab140 con 0x7fa3200ae820 2023-11-25T05:03:38.662 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa325da0640 1 librados: init done 2023-11-25T05:03:38.662 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa325da0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:38.662 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa325da0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:38.663 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa325da0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:38.663 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.657+0000 7fa325da0640 1 -- 172.21.15.90:0/2266481325 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa320075af0 con 0x7fa3200ae820 2023-11-25T05:03:38.663 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.660+0000 7fa3157fa640 1 -- 172.21.15.90:0/2266481325 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa310003c90 con 0x7fa3200ae820 2023-11-25T05:03:38.663 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.660+0000 7fa3157fa640 1 -- 172.21.15.90:0/2266481325 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fa31001c440 con 0x7fa3200ae820 2023-11-25T05:03:38.663 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.660+0000 7fa3157fa640 10 client.4114.objecter ms_dispatch 0x7fa320000b90 osd_map(3..3 src has 1..3) v4 2023-11-25T05:03:38.663 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.660+0000 7fa3157fa640 3 client.4114.objecter handle_osd_map got epochs [3,3] > 0 2023-11-25T05:03:38.663 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.660+0000 7fa3157fa640 3 client.4114.objecter handle_osd_map decoding full epoch 3 2023-11-25T05:03:38.663 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.660+0000 7fa3157fa640 20 client.4114.objecter dump_active .. 0 homeless 2023-11-25T05:03:38.682 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:38.681+0000 7f23a2432040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-25T05:03:38.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.760+0000 7fa3157fa640 1 -- 172.21.15.90:0/2266481325 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fa310017070 con 0x7fa3200ae820 2023-11-25T05:03:38.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.760+0000 7fa3157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:38.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.760+0000 7fa3157fa640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:03:38.833 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.832+0000 7fa325da0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0e252ab8-9524-4ccc-b08f-d95bbf05fbea", "id": 1}] 2023-11-25T05:03:38.833 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.832+0000 7fa325da0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0e252ab8-9524-4ccc-b08f-d95bbf05fbea", "id": 1}] 2023-11-25T05:03:38.833 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.832+0000 7fa325da0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:38.833 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.832+0000 7fa325da0640 1 -- 172.21.15.90:0/2266481325 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0e252ab8-9524-4ccc-b08f-d95bbf05fbea", "id": 1} v 0) v1 -- 0x7fa3200b3980 con 0x7fa3200ae820 2023-11-25T05:03:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.936+0000 7fa3157fa640 1 -- 172.21.15.90:0/2266481325 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0e252ab8-9524-4ccc-b08f-d95bbf05fbea", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa310011e50 con 0x7fa3200ae820 2023-11-25T05:03:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.936+0000 7fa3157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0e252ab8-9524-4ccc-b08f-d95bbf05fbea", "id": 1}] 2023-11-25T05:03:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.936+0000 7fa3157fa640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:03:38.938 INFO:teuthology.orchestra.run.smithi090.stdout:1 2023-11-25T05:03:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.937+0000 7fa325da0640 10 librados: watch_flush enter 2023-11-25T05:03:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.937+0000 7fa325da0640 10 librados: watch_flush exit 2023-11-25T05:03:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.937+0000 7fa325da0640 20 client.4114.objecter shutdown clearing up homeless session... 2023-11-25T05:03:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.937+0000 7fa325da0640 10 client.4114.objecter successfully canceled tick 2023-11-25T05:03:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.937+0000 7fa325da0640 10 monclient: shutdown 2023-11-25T05:03:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.937+0000 7fa325da0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.937+0000 7fa325da0640 1 -- 172.21.15.90:0/2266481325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3200ae820 msgr2=0x7fa3200aa130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.937+0000 7fa325da0640 1 --2- 172.21.15.90:0/2266481325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3200ae820 0x7fa3200aa130 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa31000c450 tx=0x7fa31000c910 comp rx=0 tx=0).stop 2023-11-25T05:03:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.937+0000 7fa325da0640 1 -- 172.21.15.90:0/2266481325 shutdown_connections 2023-11-25T05:03:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.937+0000 7fa325da0640 1 --2- 172.21.15.90:0/2266481325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3200b0170 0x7fa3201d6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.937+0000 7fa325da0640 1 --2- 172.21.15.90:0/2266481325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3200ae820 0x7fa3200aa130 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.937+0000 7fa325da0640 1 --2- 172.21.15.90:0/2266481325 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa32006fd50 0x7fa320139a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.938+0000 7fa325da0640 1 -- 172.21.15.90:0/2266481325 >> 172.21.15.90:0/2266481325 conn(0x7fa32006bf90 msgr2=0x7fa3200b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.938+0000 7fa325da0640 1 -- 172.21.15.90:0/2266481325 shutdown_connections 2023-11-25T05:03:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.938+0000 7fa325da0640 1 -- 172.21.15.90:0/2266481325 wait complete. 2023-11-25T05:03:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:38.938+0000 7fa325da0640 1 librados: shutdown 2023-11-25T05:03:38.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new ccda96b8-aa9e-4c99-8d84-bba19ca0483b 2 2023-11-25T05:03:39.008 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:39.007+0000 7fdccda30040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-25T05:03:39.050 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.048+0000 7ff5fa20e640 10 monclient: get_monmap_and_config 2023-11-25T05:03:39.050 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.048+0000 7ff5fa20e640 10 monclient: build_initial_monmap 2023-11-25T05:03:39.050 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.048+0000 7ff5fa20e640 10 monclient: monmap: 2023-11-25T05:03:39.050 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:39.050 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:39.050 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:03:39.049588+0000 2023-11-25T05:03:39.050 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:03:39.049588+0000 2023-11-25T05:03:39.050 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:39.050 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:39.051 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:03:39.051 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:03:39.051 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:03:39.051 INFO:teuthology.orchestra.run.smithi090.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:03:39.051 INFO:teuthology.orchestra.run.smithi090.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:03:39.051 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:39.053 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.049+0000 7ff5fa20e640 1 Processor -- start 2023-11-25T05:03:39.053 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.049+0000 7ff5fa20e640 1 -- start start 2023-11-25T05:03:39.053 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.049+0000 7ff5fa20e640 10 monclient: init 2023-11-25T05:03:39.053 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.050+0000 7ff5fa20e640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:39.053 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.050+0000 7ff5fa20e640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-11-25T05:03:39.053 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.050+0000 7ff5fa20e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f40a8a00 0x7ff5f40a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:39.053 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.050+0000 7ff5fa20e640 10 monclient(hunting): picked mon.noname-b con 0x7ff5f40a8a00 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:39.053 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.050+0000 7ff5fa20e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff5f40b2260 0x7ff5f41425d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:39.054 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.050+0000 7ff5fa20e640 10 monclient(hunting): picked mon.noname-a con 0x7ff5f40b2260 addr v2:172.21.15.7:3300/0 2023-11-25T05:03:39.054 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.050+0000 7ff5fa20e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5f4142b10 addr v1:172.21.15.7:6789/0 2023-11-25T05:03:39.054 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.050+0000 7ff5fa20e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f40aa970 con 0x7ff5f4142b10 2023-11-25T05:03:39.054 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.050+0000 7ff5fa20e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:39.054 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.050+0000 7ff5fa20e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f41421b0 con 0x7ff5f40b2260 2023-11-25T05:03:39.054 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.050+0000 7ff5fa20e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:39.054 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.050+0000 7ff5fa20e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40aac50 con 0x7ff5f40a8a00 2023-11-25T05:03:39.054 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.050+0000 7ff5fa20e640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:39.054 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.050+0000 7ff5fa20e640 10 monclient(hunting): authenticate will time out at 1320.416658s 2023-11-25T05:03:39.054 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.050+0000 7ff5f37fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f40a8a00 0x7ff5f40a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:39.054 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.051+0000 7ff5f2ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff5f40b2260 0x7ff5f41425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:39.054 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.051+0000 7ff5f37fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f40a8a00 0x7ff5f40a8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.90:60862/0 (socket says 172.21.15.90:60862) 2023-11-25T05:03:39.054 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.051+0000 7ff5f37fe640 1 -- 172.21.15.90:0/77893484 learned_addr learned my addr 172.21.15.90:0/77893484 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-11-25T05:03:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.051+0000 7ff5f37fe640 10 monclient(hunting): get_auth_request con 0x7ff5f40a8a00 auth_method 0 2023-11-25T05:03:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.051+0000 7ff5f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.051+0000 7ff5f37fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.051+0000 7ff5f37fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.051+0000 7ff5f2ffd640 10 monclient(hunting): get_auth_request con 0x7ff5f40b2260 auth_method 0 2023-11-25T05:03:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.051+0000 7ff5f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.051+0000 7ff5f2ffd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.051+0000 7ff5f2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.051+0000 7ff5f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.051+0000 7ff5f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.051+0000 7ff5f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.052+0000 7ff5f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.052+0000 7ff5f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:39.056 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.052+0000 7ff5f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:39.056 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.053+0000 7ff5f37fe640 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2023-11-25T05:03:39.056 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.053+0000 7ff5f37fe640 1 -- 172.21.15.90:0/77893484 >> v2:172.21.15.7:3300/0 conn(0x7ff5f40b2260 msgr2=0x7ff5f41425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:39.056 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.053+0000 7ff5f37fe640 1 --2- 172.21.15.90:0/77893484 >> v2:172.21.15.7:3300/0 conn(0x7ff5f40b2260 0x7ff5f41425d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.057 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.053+0000 7ff5f37fe640 1 -- 172.21.15.90:0/77893484 >> v1:172.21.15.7:6789/0 conn(0x7ff5f4142b10 legacy=0x7ff5f413f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:39.057 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.053+0000 7ff5f37fe640 10 monclient: _finish_hunting 0 2023-11-25T05:03:39.057 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.053+0000 7ff5f37fe640 1 monclient: found mon.noname-b 2023-11-25T05:03:39.057 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.053+0000 7ff5f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:39.057 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.053+0000 7ff5f37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:03:39.057 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.053+0000 7ff5f37fe640 1 -- 172.21.15.90:0/77893484 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f40aaf30 con 0x7ff5f40a8a00 2023-11-25T05:03:39.057 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.055+0000 7ff5f37fe640 1 --2- 172.21.15.90:0/77893484 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f40a8a00 0x7ff5f40a8de0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc009870 tx=0x7ff5dc02b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc90ab4489796481 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:39.060 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5f27fc640 1 -- 172.21.15.90:0/77893484 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc031740 con 0x7ff5f40a8a00 2023-11-25T05:03:39.060 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:39.060 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5f27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:03:39.060 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5f27fc640 10 monclient: dump: 2023-11-25T05:03:39.060 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:39.060 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:39.061 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5f27fc640 10 monclient: _finish_auth 0 2023-11-25T05:03:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5f27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:03:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5f27fc640 1 -- 172.21.15.90:0/77893484 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5dc0318a0 con 0x7ff5f40a8a00 2023-11-25T05:03:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5f27fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5f27fc640 1 -- 172.21.15.90:0/77893484 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc031c30 con 0x7ff5f40a8a00 2023-11-25T05:03:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5fa20e640 5 monclient: authenticate success, global_id 4112 2023-11-25T05:03:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5fa20e640 10 monclient: get_monmap_and_config success 2023-11-25T05:03:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5fa20e640 4 set_mon_vals no callback set 2023-11-25T05:03:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:03:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5f27fc640 10 monclient: dump: 2023-11-25T05:03:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:39.063 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:39.063 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:39.063 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:39.066 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5fa20e640 10 monclient: shutdown 2023-11-25T05:03:39.066 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5fa20e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:39.066 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5fa20e640 1 -- 172.21.15.90:0/77893484 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f40a8a00 msgr2=0x7ff5f40a8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:39.066 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5fa20e640 1 --2- 172.21.15.90:0/77893484 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f40a8a00 0x7ff5f40a8de0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc009870 tx=0x7ff5dc02b7e0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.066 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5fa20e640 1 -- 172.21.15.90:0/77893484 shutdown_connections 2023-11-25T05:03:39.066 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5fa20e640 1 --2- 172.21.15.90:0/77893484 >> v2:172.21.15.7:3300/0 conn(0x7ff5f40b2260 0x7ff5f41425d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.066 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5fa20e640 1 --2- 172.21.15.90:0/77893484 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f40a8a00 0x7ff5f40a8de0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.066 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5fa20e640 1 -- 172.21.15.90:0/77893484 >> 172.21.15.90:0/77893484 conn(0x7ff5f4061a90 msgr2=0x7ff5f40b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:39.066 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5fa20e640 1 -- 172.21.15.90:0/77893484 shutdown_connections 2023-11-25T05:03:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.059+0000 7ff5fa20e640 1 -- 172.21.15.90:0/77893484 wait complete. 2023-11-25T05:03:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.060+0000 7ff5fa20e640 10 monclient: build_initial_monmap 2023-11-25T05:03:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.060+0000 7ff5fa20e640 10 monclient: monmap: 2023-11-25T05:03:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:03:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2023-11-25T05:03:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2023-11-25T05:03:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:03:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:03:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:03:39.067 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.060+0000 7ff5fa20e640 1 librados: starting msgr at 2023-11-25T05:03:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.060+0000 7ff5fa20e640 1 librados: starting objecter 2023-11-25T05:03:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.060+0000 7ff5fa20e640 1 Processor -- start 2023-11-25T05:03:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 1 -- start start 2023-11-25T05:03:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 1 librados: setting wanted keys 2023-11-25T05:03:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 1 librados: calling monclient init 2023-11-25T05:03:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 10 monclient: init 2023-11-25T05:03:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-25T05:03:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5f40a8a00 0x7ff5f413c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 10 monclient(hunting): picked mon.noname-b con 0x7ff5f40a8a00 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f40b2260 0x7ff5f40b6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 10 monclient(hunting): picked mon.noname-c con 0x7ff5f40b2260 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5f4142b10 0x7ff5f40b71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 10 monclient(hunting): picked mon.noname-a con 0x7ff5f4142b10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40b63f0 con 0x7ff5f4142b10 2023-11-25T05:03:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40b66d0 con 0x7ff5f40a8a00 2023-11-25T05:03:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40b7bd0 con 0x7ff5f40b2260 2023-11-25T05:03:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.061+0000 7ff5fa20e640 10 monclient(hunting): authenticate will time out at 1320.428249s 2023-11-25T05:03:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.062+0000 7ff5f2ffd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f40b2260 0x7ff5f40b6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.062+0000 7ff5f37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5f40a8a00 0x7ff5f413c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.062+0000 7ff5f2ffd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f40b2260 0x7ff5f40b6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.90:60878/0 (socket says 172.21.15.90:60878) 2023-11-25T05:03:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.062+0000 7ff5f2ffd640 1 -- 172.21.15.90:0/1161964178 learned_addr learned my addr 172.21.15.90:0/1161964178 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-11-25T05:03:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.062+0000 7ff5f2ffd640 10 monclient(hunting): get_auth_request con 0x7ff5f40b2260 auth_method 0 2023-11-25T05:03:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.062+0000 7ff5f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.062+0000 7ff5f2ffd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.062+0000 7ff5f2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.062+0000 7ff5f37fe640 10 monclient(hunting): get_auth_request con 0x7ff5f40a8a00 auth_method 0 2023-11-25T05:03:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.062+0000 7ff5f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.062+0000 7ff5f37fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.062+0000 7ff5f37fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.062+0000 7ff5f3fff640 1 --2- 172.21.15.90:0/1161964178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5f4142b10 0x7ff5f40b71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.063+0000 7ff5f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.063+0000 7ff5f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.063+0000 7ff5f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.063+0000 7ff5f3fff640 10 monclient(hunting): get_auth_request con 0x7ff5f4142b10 auth_method 0 2023-11-25T05:03:39.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.063+0000 7ff5f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:39.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.063+0000 7ff5f3fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:39.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.063+0000 7ff5f3fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:39.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.063+0000 7ff5f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:39.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.063+0000 7ff5f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:39.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.063+0000 7ff5f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:39.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.063+0000 7ff5f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:39.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.063+0000 7ff5f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:39.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.063+0000 7ff5f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:39.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.065+0000 7ff5f2ffd640 10 monclient(hunting): handle_auth_done global_id 4115 payload 824 2023-11-25T05:03:39.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.065+0000 7ff5f2ffd640 1 -- 172.21.15.90:0/1161964178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5f40a8a00 msgr2=0x7ff5f413c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:39.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.065+0000 7ff5f2ffd640 1 --2- 172.21.15.90:0/1161964178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5f40a8a00 0x7ff5f413c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.065+0000 7ff5f2ffd640 1 -- 172.21.15.90:0/1161964178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5f4142b10 msgr2=0x7ff5f40b71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:39.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.065+0000 7ff5f2ffd640 1 --2- 172.21.15.90:0/1161964178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5f4142b10 0x7ff5f40b71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.065+0000 7ff5f2ffd640 10 monclient: _finish_hunting 0 2023-11-25T05:03:39.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.065+0000 7ff5f2ffd640 1 monclient: found mon.noname-c 2023-11-25T05:03:39.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.065+0000 7ff5f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:39.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.065+0000 7ff5f2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:03:39.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.065+0000 7ff5f2ffd640 1 -- 172.21.15.90:0/1161964178 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc0094d0 con 0x7ff5f40b2260 2023-11-25T05:03:39.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.068+0000 7ff5f2ffd640 1 --2- 172.21.15.90:0/1161964178 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f40b2260 0x7ff5f40b6c80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff5e400e980 tx=0x7ff5e400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:39.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.068+0000 7ff5ebfff640 10 client.?.objecter ms_handle_connect 0x7ff5f40b2260 2023-11-25T05:03:39.073 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.068+0000 7ff5ebfff640 10 client.?.objecter resend_mon_ops 2023-11-25T05:03:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5ebfff640 1 -- 172.21.15.90:0/1161964178 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4009670 con 0x7ff5f40b2260 2023-11-25T05:03:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5ebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:03:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5ebfff640 10 monclient: dump: 2023-11-25T05:03:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:39.075 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5ebfff640 10 monclient: _finish_auth 0 2023-11-25T05:03:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5ebfff640 10 monclient: _check_auth_tickets 2023-11-25T05:03:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5ebfff640 1 -- 172.21.15.90:0/1161964178 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e400ca00 con 0x7ff5f40b2260 2023-11-25T05:03:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5fa20e640 5 monclient: authenticate success, global_id 4115 2023-11-25T05:03:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5fa20e640 10 monclient: _renew_subs 2023-11-25T05:03:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5fa20e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:03:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5fa20e640 1 -- 172.21.15.90:0/1161964178 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f4144640 con 0x7ff5f40b2260 2023-11-25T05:03:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5ebfff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5ebfff640 1 -- 172.21.15.90:0/1161964178 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e400cd90 con 0x7ff5f40b2260 2023-11-25T05:03:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:03:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5ebfff640 10 monclient: dump: 2023-11-25T05:03:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:39.077 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5f1ffb640 4 set_mon_vals no callback set 2023-11-25T05:03:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5fa20e640 10 client.4115.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:03:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5fa20e640 10 monclient: _renew_subs 2023-11-25T05:03:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5fa20e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:03:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5fa20e640 1 -- 172.21.15.90:0/1161964178 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f40b78e0 con 0x7ff5f40b2260 2023-11-25T05:03:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.071+0000 7ff5fa20e640 1 librados: init done 2023-11-25T05:03:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.072+0000 7ff5fa20e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.072+0000 7ff5fa20e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.072+0000 7ff5fa20e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:03:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.072+0000 7ff5fa20e640 1 -- 172.21.15.90:0/1161964178 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f4067ac0 con 0x7ff5f40b2260 2023-11-25T05:03:39.090 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:39.089+0000 7f23a2432040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-25T05:03:39.095 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.093+0000 7ff5ebfff640 1 -- 172.21.15.90:0/1161964178 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff5e4005c90 con 0x7ff5f40b2260 2023-11-25T05:03:39.095 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.093+0000 7ff5ebfff640 1 -- 172.21.15.90:0/1161964178 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7ff5e4013890 con 0x7ff5f40b2260 2023-11-25T05:03:39.095 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.093+0000 7ff5ebfff640 10 client.4115.objecter ms_dispatch 0x7ff5f4000b90 osd_map(4..4 src has 1..4) v4 2023-11-25T05:03:39.095 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.093+0000 7ff5ebfff640 3 client.4115.objecter handle_osd_map got epochs [4,4] > 0 2023-11-25T05:03:39.095 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.093+0000 7ff5ebfff640 3 client.4115.objecter handle_osd_map decoding full epoch 4 2023-11-25T05:03:39.095 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.093+0000 7ff5ebfff640 20 client.4115.objecter dump_active .. 0 homeless 2023-11-25T05:03:39.148 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:39.147+0000 7fdccda30040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-25T05:03:39.234 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:39.231+0000 7f23a2432040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-25T05:03:39.266 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:39.265+0000 7fdccda30040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-25T05:03:39.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.277+0000 7ff5ebfff640 1 -- 172.21.15.90:0/1161964178 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff5e4021060 con 0x7ff5f40b2260 2023-11-25T05:03:39.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.277+0000 7ff5ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:39.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.277+0000 7ff5ebfff640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:03:39.314 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:39.312+0000 7fdccda30040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-25T05:03:39.347 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:39.346+0000 7f23a2432040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-25T05:03:39.351 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.349+0000 7ff5fa20e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ccda96b8-aa9e-4c99-8d84-bba19ca0483b", "id": 2}] 2023-11-25T05:03:39.351 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.349+0000 7ff5fa20e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ccda96b8-aa9e-4c99-8d84-bba19ca0483b", "id": 2}] 2023-11-25T05:03:39.351 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.349+0000 7ff5fa20e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:03:39.351 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.349+0000 7ff5fa20e640 1 -- 172.21.15.90:0/1161964178 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ccda96b8-aa9e-4c99-8d84-bba19ca0483b", "id": 2} v 0) v1 -- 0x7ff5f4075b60 con 0x7ff5f40b2260 2023-11-25T05:03:39.396 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:39.394+0000 7f23a2432040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-25T05:03:39.412 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:39.411+0000 7fdccda30040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-25T05:03:39.482 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.480+0000 7ff5ebfff640 1 -- 172.21.15.90:0/1161964178 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ccda96b8-aa9e-4c99-8d84-bba19ca0483b", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff5e402a0b0 con 0x7ff5f40b2260 2023-11-25T05:03:39.482 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.480+0000 7ff5ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ccda96b8-aa9e-4c99-8d84-bba19ca0483b", "id": 2}] 2023-11-25T05:03:39.482 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.480+0000 7ff5ebfff640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:03:39.482 INFO:teuthology.orchestra.run.smithi090.stdout:2 2023-11-25T05:03:39.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 10 librados: watch_flush enter 2023-11-25T05:03:39.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 10 librados: watch_flush exit 2023-11-25T05:03:39.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 20 client.4115.objecter shutdown clearing up homeless session... 2023-11-25T05:03:39.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 10 client.4115.objecter successfully canceled tick 2023-11-25T05:03:39.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 10 monclient: shutdown 2023-11-25T05:03:39.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:39.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 1 -- 172.21.15.90:0/1161964178 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f40b2260 msgr2=0x7ff5f40b6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:39.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 1 --2- 172.21.15.90:0/1161964178 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f40b2260 0x7ff5f40b6c80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff5e400e980 tx=0x7ff5e400ee40 comp rx=0 tx=0).stop 2023-11-25T05:03:39.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 1 -- 172.21.15.90:0/1161964178 shutdown_connections 2023-11-25T05:03:39.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 1 --2- 172.21.15.90:0/1161964178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5f4142b10 0x7ff5f40b71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 1 --2- 172.21.15.90:0/1161964178 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f40b2260 0x7ff5f40b6c80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 1 --2- 172.21.15.90:0/1161964178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5f40a8a00 0x7ff5f413c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 1 -- 172.21.15.90:0/1161964178 >> 172.21.15.90:0/1161964178 conn(0x7ff5f4061a90 msgr2=0x7ff5f413ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:39.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 1 -- 172.21.15.90:0/1161964178 shutdown_connections 2023-11-25T05:03:39.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 1 -- 172.21.15.90:0/1161964178 wait complete. 2023-11-25T05:03:39.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.481+0000 7ff5fa20e640 1 librados: shutdown 2023-11-25T05:03:39.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new e4b8700a-2e28-4e69-8577-465666b1b051 3 2023-11-25T05:03:39.500 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:39.499+0000 7f23a2432040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-25T05:03:39.512 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:39.510+0000 7fdccda30040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-25T05:03:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.590+0000 7f4f8e3b0640 10 monclient: get_monmap_and_config 2023-11-25T05:03:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.590+0000 7f4f8e3b0640 10 monclient: build_initial_monmap 2023-11-25T05:03:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.590+0000 7f4f8e3b0640 10 monclient: monmap: 2023-11-25T05:03:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:03:39.591709+0000 2023-11-25T05:03:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:03:39.591709+0000 2023-11-25T05:03:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:39.593 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:39.593 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:03:39.593 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:03:39.593 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:03:39.593 INFO:teuthology.orchestra.run.smithi090.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:03:39.593 INFO:teuthology.orchestra.run.smithi090.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:03:39.593 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.591+0000 7f4f8e3b0640 1 Processor -- start 2023-11-25T05:03:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.591+0000 7f4f8e3b0640 1 -- start start 2023-11-25T05:03:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.591+0000 7f4f8e3b0640 10 monclient: init 2023-11-25T05:03:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8e3b0640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8e3b0640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-11-25T05:03:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8e3b0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f880a8a00 0x7f4f880a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8e3b0640 10 monclient(hunting): picked mon.noname-b con 0x7f4f880a8a00 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8e3b0640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4f880b2260 0x7f4f881425d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8e3b0640 10 monclient(hunting): picked mon.noname-c con 0x7f4f880b2260 addr v2:172.21.15.90:3300/0 2023-11-25T05:03:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8e3b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f88142b10 addr v1:172.21.15.7:6789/0 2023-11-25T05:03:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8e3b0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f880aa970 con 0x7f4f88142b10 2023-11-25T05:03:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8e3b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8e3b0640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f881421b0 con 0x7f4f880b2260 2023-11-25T05:03:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8e3b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8e3b0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f880aac50 con 0x7f4f880a8a00 2023-11-25T05:03:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8e3b0640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8e3b0640 10 monclient(hunting): authenticate will time out at 1320.958722s 2023-11-25T05:03:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f877fe640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4f880b2260 0x7f4f881425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f87fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f880a8a00 0x7f4f880a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8c94d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4f88142b10 0x7f4f8813f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.90:50846/0 (socket says 172.21.15.90:50846) 2023-11-25T05:03:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.592+0000 7f4f8c94d640 1 -- 172.21.15.90:0/2751251737 learned_addr learned my addr 172.21.15.90:0/2751251737 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-11-25T05:03:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.593+0000 7f4f87fff640 10 monclient(hunting): get_auth_request con 0x7f4f880a8a00 auth_method 0 2023-11-25T05:03:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.593+0000 7f4f87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.593+0000 7f4f87fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.593+0000 7f4f87fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.593+0000 7f4f877fe640 10 monclient(hunting): get_auth_request con 0x7f4f880b2260 auth_method 0 2023-11-25T05:03:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.593+0000 7f4f877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.593+0000 7f4f877fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.593+0000 7f4f877fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.593+0000 7f4f87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.593+0000 7f4f87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.593+0000 7f4f87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.594+0000 7f4f877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.594+0000 7f4f877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.594+0000 7f4f877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.595+0000 7f4f87fff640 10 monclient(hunting): handle_auth_done global_id 4118 payload 274 2023-11-25T05:03:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.595+0000 7f4f87fff640 1 -- 172.21.15.90:0/2751251737 >> v2:172.21.15.90:3300/0 conn(0x7f4f880b2260 msgr2=0x7f4f881425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:39.599 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.595+0000 7f4f87fff640 1 --2- 172.21.15.90:0/2751251737 >> v2:172.21.15.90:3300/0 conn(0x7f4f880b2260 0x7f4f881425d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.599 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.595+0000 7f4f87fff640 1 -- 172.21.15.90:0/2751251737 >> v1:172.21.15.7:6789/0 conn(0x7f4f88142b10 legacy=0x7f4f8813f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:39.599 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.595+0000 7f4f87fff640 10 monclient: _finish_hunting 0 2023-11-25T05:03:39.599 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.595+0000 7f4f87fff640 1 monclient: found mon.noname-b 2023-11-25T05:03:39.599 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.595+0000 7f4f87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:39.599 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.595+0000 7f4f87fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:03:39.599 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.595+0000 7f4f87fff640 1 -- 172.21.15.90:0/2751251737 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f880aaf30 con 0x7f4f880a8a00 2023-11-25T05:03:39.599 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.597+0000 7f4f87fff640 1 --2- 172.21.15.90:0/2751251737 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f880a8a00 0x7f4f880a8de0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c0097a0 tx=0x7f4f7c02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69dbbec48c256721 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:39.601 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:39.599+0000 7f23a2432040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-25T05:03:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f86ffd640 1 -- 172.21.15.90:0/2751251737 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c02f050 con 0x7f4f880a8a00 2023-11-25T05:03:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f86ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:03:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f86ffd640 10 monclient: dump: 2023-11-25T05:03:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:39.608 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f86ffd640 10 monclient: _finish_auth 0 2023-11-25T05:03:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f86ffd640 10 monclient: _check_auth_tickets 2023-11-25T05:03:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f86ffd640 1 -- 172.21.15.90:0/2751251737 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7c031700 con 0x7f4f880a8a00 2023-11-25T05:03:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f8e3b0640 5 monclient: authenticate success, global_id 4118 2023-11-25T05:03:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f8e3b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:03:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f86ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f8e3b0640 10 monclient: get_monmap_and_config success 2023-11-25T05:03:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f8e3b0640 4 set_mon_vals no callback set 2023-11-25T05:03:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f86ffd640 1 -- 172.21.15.90:0/2751251737 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c031a90 con 0x7f4f880a8a00 2023-11-25T05:03:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:03:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f86ffd640 10 monclient: dump: 2023-11-25T05:03:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:39.610 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f8e3b0640 10 monclient: shutdown 2023-11-25T05:03:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f8e3b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f8e3b0640 1 -- 172.21.15.90:0/2751251737 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f880a8a00 msgr2=0x7f4f880a8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.601+0000 7f4f8e3b0640 1 --2- 172.21.15.90:0/2751251737 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f880a8a00 0x7f4f880a8de0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c0097a0 tx=0x7f4f7c02b7a0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.602+0000 7f4f8e3b0640 1 -- 172.21.15.90:0/2751251737 shutdown_connections 2023-11-25T05:03:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.602+0000 7f4f8e3b0640 1 --2- 172.21.15.90:0/2751251737 >> v2:172.21.15.90:3300/0 conn(0x7f4f880b2260 0x7f4f881425d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.602+0000 7f4f8e3b0640 1 --2- 172.21.15.90:0/2751251737 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f880a8a00 0x7f4f880a8de0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.602+0000 7f4f8e3b0640 1 -- 172.21.15.90:0/2751251737 >> 172.21.15.90:0/2751251737 conn(0x7f4f88061a90 msgr2=0x7f4f880b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.602+0000 7f4f8e3b0640 1 -- 172.21.15.90:0/2751251737 shutdown_connections 2023-11-25T05:03:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.602+0000 7f4f8e3b0640 1 -- 172.21.15.90:0/2751251737 wait complete. 2023-11-25T05:03:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.602+0000 7f4f8e3b0640 10 monclient: build_initial_monmap 2023-11-25T05:03:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.602+0000 7f4f8e3b0640 10 monclient: monmap: 2023-11-25T05:03:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:03:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2023-11-25T05:03:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2023-11-25T05:03:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:03:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:03:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:03:39.612 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.602+0000 7f4f8e3b0640 1 librados: starting msgr at 2023-11-25T05:03:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.602+0000 7f4f8e3b0640 1 librados: starting objecter 2023-11-25T05:03:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 1 Processor -- start 2023-11-25T05:03:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 1 -- start start 2023-11-25T05:03:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 1 librados: setting wanted keys 2023-11-25T05:03:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 1 librados: calling monclient init 2023-11-25T05:03:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 10 monclient: init 2023-11-25T05:03:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:03:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f880a8a00 0x7f4f8813c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 10 monclient(hunting): picked mon.noname-a con 0x7f4f880a8a00 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f880b2260 0x7f4f880b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 10 monclient(hunting): picked mon.noname-c con 0x7f4f880b2260 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f88142b10 0x7f4f881da870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 10 monclient(hunting): picked mon.noname-b con 0x7f4f88142b10 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f880b5e30 con 0x7f4f880a8a00 2023-11-25T05:03:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8813b3e0 con 0x7f4f88142b10 2023-11-25T05:03:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8813bf20 con 0x7f4f880b2260 2023-11-25T05:03:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.603+0000 7f4f8e3b0640 10 monclient(hunting): authenticate will time out at 1320.970338s 2023-11-25T05:03:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.604+0000 7f4f877fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f880b2260 0x7f4f880b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.604+0000 7f4f8c94d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f88142b10 0x7f4f881da870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.604+0000 7f4f877fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f880b2260 0x7f4f880b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.90:60900/0 (socket says 172.21.15.90:60900) 2023-11-25T05:03:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.604+0000 7f4f877fe640 1 -- 172.21.15.90:0/3954483747 learned_addr learned my addr 172.21.15.90:0/3954483747 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-11-25T05:03:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.604+0000 7f4f877fe640 10 monclient(hunting): get_auth_request con 0x7f4f880b2260 auth_method 0 2023-11-25T05:03:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.604+0000 7f4f877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.604+0000 7f4f877fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.604+0000 7f4f877fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.604+0000 7f4f8c94d640 10 monclient(hunting): get_auth_request con 0x7f4f88142b10 auth_method 0 2023-11-25T05:03:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.604+0000 7f4f8c94d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.604+0000 7f4f8c94d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.604+0000 7f4f8c94d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.605+0000 7f4f877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.605+0000 7f4f877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.605+0000 7f4f877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:39.615 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.605+0000 7f4f8c94d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:39.615 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.605+0000 7f4f8c94d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:39.615 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.605+0000 7f4f8c94d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:39.617 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.608+0000 7f4f8c94d640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2023-11-25T05:03:39.617 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.608+0000 7f4f8c94d640 1 -- 172.21.15.90:0/3954483747 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f880b2260 msgr2=0x7f4f880b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:39.617 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.608+0000 7f4f8c94d640 1 --2- 172.21.15.90:0/3954483747 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f880b2260 0x7f4f880b58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.617 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.608+0000 7f4f8c94d640 1 -- 172.21.15.90:0/3954483747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f880a8a00 msgr2=0x7f4f8813c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:39.617 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.608+0000 7f4f8c94d640 1 --2- 172.21.15.90:0/3954483747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f880a8a00 0x7f4f8813c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.617 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.608+0000 7f4f8c94d640 10 monclient: _finish_hunting 0 2023-11-25T05:03:39.617 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.608+0000 7f4f8c94d640 1 monclient: found mon.noname-b 2023-11-25T05:03:39.617 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.608+0000 7f4f8c94d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:39.618 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.608+0000 7f4f8c94d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:03:39.618 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.608+0000 7f4f8c94d640 1 -- 172.21.15.90:0/3954483747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c009400 con 0x7f4f88142b10 2023-11-25T05:03:39.618 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.610+0000 7f4f8c94d640 1 --2- 172.21.15.90:0/3954483747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f88142b10 0x7f4f881da870 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000caa0 tx=0x7f4f8000cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:39.618 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.610+0000 7f4f84ff9640 10 client.?.objecter ms_handle_connect 0x7f4f88142b10 2023-11-25T05:03:39.618 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.610+0000 7f4f84ff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:03:39.618 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f84ff9640 1 -- 172.21.15.90:0/3954483747 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8000ab40 con 0x7f4f88142b10 2023-11-25T05:03:39.618 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:39.618 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f84ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:03:39.618 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f84ff9640 10 monclient: dump: 2023-11-25T05:03:39.618 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:39.618 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:39.618 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.618 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:39.619 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f84ff9640 10 monclient: _finish_auth 0 2023-11-25T05:03:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f84ff9640 10 monclient: _check_auth_tickets 2023-11-25T05:03:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f84ff9640 1 -- 172.21.15.90:0/3954483747 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8000aca0 con 0x7f4f88142b10 2023-11-25T05:03:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f84ff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f84ff9640 1 -- 172.21.15.90:0/3954483747 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f80012600 con 0x7f4f88142b10 2023-11-25T05:03:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f8e3b0640 5 monclient: authenticate success, global_id 4123 2023-11-25T05:03:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f8e3b0640 10 monclient: _renew_subs 2023-11-25T05:03:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f8e3b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f8e3b0640 1 -- 172.21.15.90:0/3954483747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f880b72a0 con 0x7f4f88142b10 2023-11-25T05:03:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f84ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:03:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f84ff9640 10 monclient: dump: 2023-11-25T05:03:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:39.621 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:39.621 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:39.621 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:39.621 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f867fc640 4 set_mon_vals no callback set 2023-11-25T05:03:39.622 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f8e3b0640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:03:39.622 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f8e3b0640 10 monclient: _renew_subs 2023-11-25T05:03:39.622 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f8e3b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:39.622 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f8e3b0640 1 -- 172.21.15.90:0/3954483747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f881daf90 con 0x7f4f88142b10 2023-11-25T05:03:39.622 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.613+0000 7f4f8e3b0640 1 librados: init done 2023-11-25T05:03:39.622 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.614+0000 7f4f8e3b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:39.622 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.614+0000 7f4f8e3b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:39.622 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.614+0000 7f4f8e3b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:39.622 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.614+0000 7f4f8e3b0640 1 -- 172.21.15.90:0/3954483747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f88067ac0 con 0x7f4f88142b10 2023-11-25T05:03:39.662 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:39.660+0000 7fdccda30040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-25T05:03:39.706 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:39.705+0000 7fdccda30040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-25T05:03:39.724 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.722+0000 7f4f84ff9640 1 -- 172.21.15.90:0/3954483747 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4f80012a60 con 0x7f4f88142b10 2023-11-25T05:03:39.724 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.722+0000 7f4f84ff9640 1 -- 172.21.15.90:0/3954483747 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f4f80007500 con 0x7f4f88142b10 2023-11-25T05:03:39.724 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.722+0000 7f4f84ff9640 10 client.4123.objecter ms_dispatch 0x7f4f88000b90 osd_map(5..5 src has 1..5) v4 2023-11-25T05:03:39.724 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.722+0000 7f4f84ff9640 3 client.4123.objecter handle_osd_map got epochs [5,5] > 0 2023-11-25T05:03:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.722+0000 7f4f84ff9640 3 client.4123.objecter handle_osd_map decoding full epoch 5 2023-11-25T05:03:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.722+0000 7f4f84ff9640 20 client.4123.objecter dump_active .. 0 homeless 2023-11-25T05:03:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.722+0000 7f4f84ff9640 1 -- 172.21.15.90:0/3954483747 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f4f80021060 con 0x7f4f88142b10 2023-11-25T05:03:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.722+0000 7f4f84ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.722+0000 7f4f84ff9640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:03:39.752 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:39.750+0000 7f23a2432040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-25T05:03:39.798 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:39.796+0000 7f23a2432040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-25T05:03:39.818 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.816+0000 7f4f8e3b0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e4b8700a-2e28-4e69-8577-465666b1b051", "id": 3}] 2023-11-25T05:03:39.818 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.816+0000 7f4f8e3b0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e4b8700a-2e28-4e69-8577-465666b1b051", "id": 3}] 2023-11-25T05:03:39.818 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.816+0000 7f4f8e3b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:39.818 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.816+0000 7f4f8e3b0640 1 -- 172.21.15.90:0/3954483747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e4b8700a-2e28-4e69-8577-465666b1b051", "id": 3} v 0) v1 -- 0x7f4f881db1e0 con 0x7f4f88142b10 2023-11-25T05:03:39.853 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:39.850+0000 7fdccda30040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-25T05:03:39.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.921+0000 7f4f84ff9640 1 -- 172.21.15.90:0/3954483747 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e4b8700a-2e28-4e69-8577-465666b1b051", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4f881db1e0 con 0x7f4f88142b10 2023-11-25T05:03:39.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.921+0000 7f4f84ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e4b8700a-2e28-4e69-8577-465666b1b051", "id": 3}] 2023-11-25T05:03:39.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.921+0000 7f4f84ff9640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:03:39.923 INFO:teuthology.orchestra.run.smithi090.stdout:3 2023-11-25T05:03:39.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.922+0000 7f4f8e3b0640 10 librados: watch_flush enter 2023-11-25T05:03:39.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.922+0000 7f4f8e3b0640 10 librados: watch_flush exit 2023-11-25T05:03:39.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.922+0000 7f4f8e3b0640 20 client.4123.objecter shutdown clearing up homeless session... 2023-11-25T05:03:39.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.922+0000 7f4f8e3b0640 10 client.4123.objecter successfully canceled tick 2023-11-25T05:03:39.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.922+0000 7f4f8e3b0640 10 monclient: shutdown 2023-11-25T05:03:39.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.922+0000 7f4f8e3b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:39.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.922+0000 7f4f8e3b0640 1 -- 172.21.15.90:0/3954483747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f88142b10 msgr2=0x7f4f881da870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:39.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.922+0000 7f4f8e3b0640 1 --2- 172.21.15.90:0/3954483747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f88142b10 0x7f4f881da870 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000caa0 tx=0x7f4f8000cf60 comp rx=0 tx=0).stop 2023-11-25T05:03:39.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.922+0000 7f4f8e3b0640 1 -- 172.21.15.90:0/3954483747 shutdown_connections 2023-11-25T05:03:39.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.922+0000 7f4f8e3b0640 1 --2- 172.21.15.90:0/3954483747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f88142b10 0x7f4f881da870 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.922+0000 7f4f8e3b0640 1 --2- 172.21.15.90:0/3954483747 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f880b2260 0x7f4f880b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.926 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.922+0000 7f4f8e3b0640 1 --2- 172.21.15.90:0/3954483747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f880a8a00 0x7f4f8813c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:39.926 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.922+0000 7f4f8e3b0640 1 -- 172.21.15.90:0/3954483747 >> 172.21.15.90:0/3954483747 conn(0x7f4f88061a90 msgr2=0x7f4f88138f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:39.926 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.922+0000 7f4f8e3b0640 1 -- 172.21.15.90:0/3954483747 shutdown_connections 2023-11-25T05:03:39.926 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.922+0000 7f4f8e3b0640 1 -- 172.21.15.90:0/3954483747 wait complete. 2023-11-25T05:03:39.926 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:39.923+0000 7f4f8e3b0640 1 librados: shutdown 2023-11-25T05:03:39.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 36c88a76-0467-4e37-85b4-dfd5953855a3 4 2023-11-25T05:03:39.954 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:39.952+0000 7f23a2432040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-25T05:03:40.073 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.072+0000 7f79ea9fb640 10 monclient: get_monmap_and_config 2023-11-25T05:03:40.074 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.072+0000 7f79ea9fb640 10 monclient: build_initial_monmap 2023-11-25T05:03:40.074 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.072+0000 7f79ea9fb640 10 monclient: monmap: 2023-11-25T05:03:40.074 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:40.074 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:40.074 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:03:40.073105+0000 2023-11-25T05:03:40.074 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:03:40.073105+0000 2023-11-25T05:03:40.074 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:03:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:03:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:03:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:03:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:03:40.075 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.072+0000 7f79ea9fb640 1 Processor -- start 2023-11-25T05:03:40.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.072+0000 7f79ea9fb640 1 -- start start 2023-11-25T05:03:40.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.072+0000 7f79ea9fb640 10 monclient: init 2023-11-25T05:03:40.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.073+0000 7f79ea9fb640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:40.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.073+0000 7f79ea9fb640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2023-11-25T05:03:40.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.073+0000 7f79ea9fb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79e40a8a00 addr v1:172.21.15.7:6789/0 2023-11-25T05:03:40.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.073+0000 7f79ea9fb640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f79e40b2260 0x7f79e40a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:40.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.073+0000 7f79ea9fb640 10 monclient(hunting): picked mon.noname-c con 0x7f79e40b2260 addr v2:172.21.15.90:3300/0 2023-11-25T05:03:40.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.073+0000 7f79ea9fb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79e413f080 addr v1:172.21.15.90:6789/0 2023-11-25T05:03:40.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.073+0000 7f79ea9fb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e40aa970 con 0x7f79e40a8a00 2023-11-25T05:03:40.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.073+0000 7f79ea9fb640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e40aac50 con 0x7f79e413f080 2023-11-25T05:03:40.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.073+0000 7f79ea9fb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:40.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.073+0000 7f79ea9fb640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e41421b0 con 0x7f79e40b2260 2023-11-25T05:03:40.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.073+0000 7f79ea9fb640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:40.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.073+0000 7f79ea9fb640 10 monclient(hunting): authenticate will time out at 1321.439979s 2023-11-25T05:03:40.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.074+0000 7f79e37fe640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f79e40b2260 0x7f79e40a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:40.078 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.074+0000 7f79e3fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f79e40a8a00 0x7f79e41425d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.90:50854/0 (socket says 172.21.15.90:50854) 2023-11-25T05:03:40.078 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.074+0000 7f79e3fff640 1 -- 172.21.15.90:0/1527626097 learned_addr learned my addr 172.21.15.90:0/1527626097 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-11-25T05:03:40.078 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.074+0000 7f79e37fe640 10 monclient(hunting): get_auth_request con 0x7f79e40b2260 auth_method 0 2023-11-25T05:03:40.078 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.074+0000 7f79e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:40.078 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.074+0000 7f79e37fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.078 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.074+0000 7f79e37fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.079 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.075+0000 7f79e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:40.079 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.075+0000 7f79e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:40.079 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.075+0000 7f79e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:40.079 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.077+0000 7f79e2ffd640 1 -- 172.21.15.90:0/1527626097 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901639186 0 0) 0x7f79e40aa970 con 0x7f79e40a8a00 2023-11-25T05:03:40.080 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.077+0000 7f79e2ffd640 10 monclient(hunting): _negotiate 2023-11-25T05:03:40.080 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.077+0000 7f79e2ffd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.080 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.077+0000 7f79e2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.081 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.077+0000 7f79e2ffd640 10 monclient(hunting): my global_id is 4131 2023-11-25T05:03:40.081 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.077+0000 7f79e2ffd640 1 -- 172.21.15.90:0/1527626097 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79cc003520 con 0x7f79e40a8a00 2023-11-25T05:03:40.081 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.079+0000 7f79e37fe640 10 monclient(hunting): handle_auth_done global_id 4126 payload 274 2023-11-25T05:03:40.082 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.079+0000 7f79e37fe640 1 -- 172.21.15.90:0/1527626097 >> v1:172.21.15.90:6789/0 conn(0x7f79e413f080 legacy=0x7f79e413f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.082 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.079+0000 7f79e37fe640 1 -- 172.21.15.90:0/1527626097 >> v1:172.21.15.7:6789/0 conn(0x7f79e40a8a00 legacy=0x7f79e41425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.082 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.079+0000 7f79e37fe640 10 monclient: _finish_hunting 0 2023-11-25T05:03:40.082 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.079+0000 7f79e37fe640 1 monclient: found mon.noname-c 2023-11-25T05:03:40.082 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.079+0000 7f79e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:40.082 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.079+0000 7f79e37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.082 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.079+0000 7f79e37fe640 1 -- 172.21.15.90:0/1527626097 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e40aaf30 con 0x7f79e40b2260 2023-11-25T05:03:40.085 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.084+0000 7f79e37fe640 1 --2- 172.21.15.90:0/1527626097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e40b2260 0x7f79e40a8de0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f79d80097a0 tx=0x7f79d802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dacda0f278d00e86 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:40.090 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79e2ffd640 1 -- 172.21.15.90:0/1527626097 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d802e070 con 0x7f79e40b2260 2023-11-25T05:03:40.090 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:40.091 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:03:40.091 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79e2ffd640 10 monclient: dump: 2023-11-25T05:03:40.091 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:40.091 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:40.091 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.091 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.091 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:40.091 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.091 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:40.091 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:40.091 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:40.091 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.091 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79e2ffd640 10 monclient: _finish_auth 0 2023-11-25T05:03:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79e2ffd640 10 monclient: _check_auth_tickets 2023-11-25T05:03:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79ea9fb640 5 monclient: authenticate success, global_id 4126 2023-11-25T05:03:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79ea9fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:03:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79e2ffd640 1 -- 172.21.15.90:0/1527626097 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d80315a0 con 0x7f79e40b2260 2023-11-25T05:03:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79e2ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79ea9fb640 10 monclient: get_monmap_and_config success 2023-11-25T05:03:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79ea9fb640 4 set_mon_vals no callback set 2023-11-25T05:03:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79e2ffd640 1 -- 172.21.15.90:0/1527626097 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d8031930 con 0x7f79e40b2260 2023-11-25T05:03:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:03:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79e2ffd640 10 monclient: dump: 2023-11-25T05:03:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:40.093 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:40.093 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.093 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.093 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:40.093 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.093 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:40.093 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:40.093 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:40.093 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.097 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79ea9fb640 10 monclient: shutdown 2023-11-25T05:03:40.097 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79ea9fb640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79ea9fb640 1 -- 172.21.15.90:0/1527626097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e40b2260 msgr2=0x7f79e40a8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79ea9fb640 1 --2- 172.21.15.90:0/1527626097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e40b2260 0x7f79e40a8de0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f79d80097a0 tx=0x7f79d802b840 comp rx=0 tx=0).stop 2023-11-25T05:03:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79ea9fb640 1 -- 172.21.15.90:0/1527626097 shutdown_connections 2023-11-25T05:03:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79ea9fb640 1 --2- 172.21.15.90:0/1527626097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e40b2260 0x7f79e40a8de0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79ea9fb640 1 -- 172.21.15.90:0/1527626097 >> 172.21.15.90:0/1527626097 conn(0x7f79e4061a90 msgr2=0x7f79e40b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79ea9fb640 1 -- 172.21.15.90:0/1527626097 shutdown_connections 2023-11-25T05:03:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.089+0000 7f79ea9fb640 1 -- 172.21.15.90:0/1527626097 wait complete. 2023-11-25T05:03:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.090+0000 7f79ea9fb640 10 monclient: build_initial_monmap 2023-11-25T05:03:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.090+0000 7f79ea9fb640 10 monclient: monmap: 2023-11-25T05:03:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:03:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2023-11-25T05:03:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2023-11-25T05:03:40.099 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:40.099 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.099 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:03:40.099 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:03:40.099 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:03:40.099 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.099 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.090+0000 7f79ea9fb640 1 librados: starting msgr at 2023-11-25T05:03:40.099 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.090+0000 7f79ea9fb640 1 librados: starting objecter 2023-11-25T05:03:40.099 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.090+0000 7f79ea9fb640 1 Processor -- start 2023-11-25T05:03:40.099 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 1 -- start start 2023-11-25T05:03:40.099 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 1 librados: setting wanted keys 2023-11-25T05:03:40.099 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 1 librados: calling monclient init 2023-11-25T05:03:40.099 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 10 monclient: init 2023-11-25T05:03:40.099 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-25T05:03:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e40a8a00 0x7f79e41d7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 10 monclient(hunting): picked mon.noname-b con 0x7f79e40a8a00 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79e40b2260 0x7f79e4143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 10 monclient(hunting): picked mon.noname-c con 0x7f79e40b2260 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e413f080 0x7f79e41d7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 10 monclient(hunting): picked mon.noname-a con 0x7f79e413f080 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e413fc40 con 0x7f79e413f080 2023-11-25T05:03:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e4143840 con 0x7f79e40a8a00 2023-11-25T05:03:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e41360b0 con 0x7f79e40b2260 2023-11-25T05:03:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.091+0000 7f79ea9fb640 10 monclient(hunting): authenticate will time out at 1321.458400s 2023-11-25T05:03:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e40a8a00 0x7f79e41d7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e8f98640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e413f080 0x7f79e41d7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e37fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79e40b2260 0x7f79e4143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e40a8a00 0x7f79e41d7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40410/0 (socket says 172.21.15.90:40410) 2023-11-25T05:03:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e3fff640 1 -- 172.21.15.90:0/4256639424 learned_addr learned my addr 172.21.15.90:0/4256639424 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-11-25T05:03:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e3fff640 10 monclient(hunting): get_auth_request con 0x7f79e40a8a00 auth_method 0 2023-11-25T05:03:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e3fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e3fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e37fe640 10 monclient(hunting): get_auth_request con 0x7f79e40b2260 auth_method 0 2023-11-25T05:03:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e37fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e37fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e8f98640 10 monclient(hunting): get_auth_request con 0x7f79e413f080 auth_method 0 2023-11-25T05:03:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e8f98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e8f98640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.092+0000 7f79e8f98640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.093+0000 7f79e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.093+0000 7f79e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.093+0000 7f79e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.093+0000 7f79e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.093+0000 7f79e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.093+0000 7f79e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.093+0000 7f79e8f98640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.093+0000 7f79e8f98640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.094+0000 7f79e8f98640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.095+0000 7f79e3fff640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2023-11-25T05:03:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.095+0000 7f79e3fff640 1 -- 172.21.15.90:0/4256639424 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79e40b2260 msgr2=0x7f79e4143300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.095+0000 7f79e3fff640 1 --2- 172.21.15.90:0/4256639424 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79e40b2260 0x7f79e4143300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.095+0000 7f79e3fff640 1 -- 172.21.15.90:0/4256639424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e413f080 msgr2=0x7f79e41d7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.095+0000 7f79e3fff640 1 --2- 172.21.15.90:0/4256639424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e413f080 0x7f79e41d7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.095+0000 7f79e3fff640 10 monclient: _finish_hunting 0 2023-11-25T05:03:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.095+0000 7f79e3fff640 1 monclient: found mon.noname-b 2023-11-25T05:03:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.095+0000 7f79e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.095+0000 7f79e3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.095+0000 7f79e3fff640 1 -- 172.21.15.90:0/4256639424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d8009400 con 0x7f79e40a8a00 2023-11-25T05:03:40.104 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.098+0000 7f79e3fff640 1 --2- 172.21.15.90:0/4256639424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e40a8a00 0x7f79e41d7290 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f79d400c500 tx=0x7f79d400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.098+0000 7f79e0ff9640 10 client.?.objecter ms_handle_connect 0x7f79e40a8a00 2023-11-25T05:03:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.098+0000 7f79e0ff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:03:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79e0ff9640 1 -- 172.21.15.90:0/4256639424 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d4005ce0 con 0x7f79e40a8a00 2023-11-25T05:03:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79e0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:03:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79e0ff9640 10 monclient: dump: 2023-11-25T05:03:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:40.106 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79e0ff9640 10 monclient: _finish_auth 0 2023-11-25T05:03:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79e0ff9640 10 monclient: _check_auth_tickets 2023-11-25T05:03:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79ea9fb640 5 monclient: authenticate success, global_id 4132 2023-11-25T05:03:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79e0ff9640 1 -- 172.21.15.90:0/4256639424 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d4005e40 con 0x7f79e40a8a00 2023-11-25T05:03:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79e0ff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79e0ff9640 1 -- 172.21.15.90:0/4256639424 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d4007330 con 0x7f79e40a8a00 2023-11-25T05:03:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79ea9fb640 10 monclient: _renew_subs 2023-11-25T05:03:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79ea9fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79ea9fb640 1 -- 172.21.15.90:0/4256639424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e41347e0 con 0x7f79e40a8a00 2023-11-25T05:03:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:03:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79e0ff9640 10 monclient: dump: 2023-11-25T05:03:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:40.108 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79e27fc640 4 set_mon_vals no callback set 2023-11-25T05:03:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79ea9fb640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:03:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79ea9fb640 10 monclient: _renew_subs 2023-11-25T05:03:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79ea9fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79ea9fb640 1 -- 172.21.15.90:0/4256639424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e41d7e50 con 0x7f79e40a8a00 2023-11-25T05:03:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.101+0000 7f79ea9fb640 1 librados: init done 2023-11-25T05:03:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.102+0000 7f79ea9fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.102+0000 7f79ea9fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.102+0000 7f79ea9fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.102+0000 7f79ea9fb640 1 -- 172.21.15.90:0/4256639424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e4067ac0 con 0x7f79e40a8a00 2023-11-25T05:03:40.206 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.204+0000 7f79e0ff9640 1 -- 172.21.15.90:0/4256639424 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f79d401c440 con 0x7f79e40a8a00 2023-11-25T05:03:40.206 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.205+0000 7f79e0ff9640 1 -- 172.21.15.90:0/4256639424 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f79d401b8c0 con 0x7f79e40a8a00 2023-11-25T05:03:40.206 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.205+0000 7f79e0ff9640 10 client.4132.objecter ms_dispatch 0x7f79e4000b90 osd_map(6..6 src has 1..6) v4 2023-11-25T05:03:40.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.205+0000 7f79e0ff9640 3 client.4132.objecter handle_osd_map got epochs [6,6] > 0 2023-11-25T05:03:40.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.205+0000 7f79e0ff9640 3 client.4132.objecter handle_osd_map decoding full epoch 6 2023-11-25T05:03:40.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.205+0000 7f79e0ff9640 20 client.4132.objecter dump_active .. 0 homeless 2023-11-25T05:03:40.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.205+0000 7f79e0ff9640 1 -- 172.21.15.90:0/4256639424 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f79d4017070 con 0x7f79e40a8a00 2023-11-25T05:03:40.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.205+0000 7f79e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:40.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.205+0000 7f79e0ff9640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:03:40.278 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.276+0000 7f79ea9fb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "36c88a76-0467-4e37-85b4-dfd5953855a3", "id": 4}] 2023-11-25T05:03:40.278 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.276+0000 7f79ea9fb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "36c88a76-0467-4e37-85b4-dfd5953855a3", "id": 4}] 2023-11-25T05:03:40.278 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.276+0000 7f79ea9fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.278 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.276+0000 7f79ea9fb640 1 -- 172.21.15.90:0/4256639424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd new", "uuid": "36c88a76-0467-4e37-85b4-dfd5953855a3", "id": 4} v 0) v1 -- 0x7f79e4134d00 con 0x7f79e40a8a00 2023-11-25T05:03:40.342 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.340+0000 7f79e0ff9640 1 -- 172.21.15.90:0/4256639424 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "36c88a76-0467-4e37-85b4-dfd5953855a3", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f79e4134d00 con 0x7f79e40a8a00 2023-11-25T05:03:40.342 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.341+0000 7f79e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "36c88a76-0467-4e37-85b4-dfd5953855a3", "id": 4}] 2023-11-25T05:03:40.342 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.341+0000 7f79e0ff9640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:03:40.343 INFO:teuthology.orchestra.run.smithi090.stdout:4 2023-11-25T05:03:40.344 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.341+0000 7f79ea9fb640 10 librados: watch_flush enter 2023-11-25T05:03:40.344 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.341+0000 7f79ea9fb640 10 librados: watch_flush exit 2023-11-25T05:03:40.344 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.341+0000 7f79ea9fb640 20 client.4132.objecter shutdown clearing up homeless session... 2023-11-25T05:03:40.344 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.341+0000 7f79ea9fb640 10 client.4132.objecter successfully canceled tick 2023-11-25T05:03:40.344 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.341+0000 7f79ea9fb640 10 monclient: shutdown 2023-11-25T05:03:40.344 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.341+0000 7f79ea9fb640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:40.344 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.341+0000 7f79ea9fb640 1 -- 172.21.15.90:0/4256639424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e40a8a00 msgr2=0x7f79e41d7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.344 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.341+0000 7f79ea9fb640 1 --2- 172.21.15.90:0/4256639424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e40a8a00 0x7f79e41d7290 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f79d400c500 tx=0x7f79d400c9c0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.345 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.341+0000 7f79ea9fb640 1 -- 172.21.15.90:0/4256639424 shutdown_connections 2023-11-25T05:03:40.345 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.342+0000 7f79ea9fb640 1 --2- 172.21.15.90:0/4256639424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e413f080 0x7f79e41d7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.345 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.342+0000 7f79ea9fb640 1 --2- 172.21.15.90:0/4256639424 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79e40b2260 0x7f79e4143300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.345 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.342+0000 7f79ea9fb640 1 --2- 172.21.15.90:0/4256639424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e40a8a00 0x7f79e41d7290 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.345 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.342+0000 7f79ea9fb640 1 -- 172.21.15.90:0/4256639424 >> 172.21.15.90:0/4256639424 conn(0x7f79e4061a90 msgr2=0x7f79e413f510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:40.345 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.342+0000 7f79ea9fb640 1 -- 172.21.15.90:0/4256639424 shutdown_connections 2023-11-25T05:03:40.345 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.342+0000 7f79ea9fb640 1 -- 172.21.15.90:0/4256639424 wait complete. 2023-11-25T05:03:40.345 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.342+0000 7f79ea9fb640 1 librados: shutdown 2023-11-25T05:03:40.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new ebf452a0-1b17-488a-bbea-2e953c932ac7 5 2023-11-25T05:03:40.454 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.452+0000 7f26a781f640 10 monclient: get_monmap_and_config 2023-11-25T05:03:40.454 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.452+0000 7f26a781f640 10 monclient: build_initial_monmap 2023-11-25T05:03:40.455 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.452+0000 7f26a781f640 10 monclient: monmap: 2023-11-25T05:03:40.455 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:40.455 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:40.455 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:03:40.453777+0000 2023-11-25T05:03:40.456 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:03:40.453777+0000 2023-11-25T05:03:40.456 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:40.456 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.456 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:03:40.457 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:03:40.457 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:03:40.457 INFO:teuthology.orchestra.run.smithi090.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:03:40.457 INFO:teuthology.orchestra.run.smithi090.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:03:40.458 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.461 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.453+0000 7f26a781f640 1 Processor -- start 2023-11-25T05:03:40.461 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.453+0000 7f26a781f640 1 -- start start 2023-11-25T05:03:40.461 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.453+0000 7f26a781f640 10 monclient: init 2023-11-25T05:03:40.462 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a781f640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:40.462 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a781f640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-11-25T05:03:40.462 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a781f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f26a00a8a00 0x7f26a00a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:40.462 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a781f640 10 monclient(hunting): picked mon.noname-b con 0x7f26a00a8a00 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:40.463 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a781f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26a00b2260 addr v1:172.21.15.90:6789/0 2023-11-25T05:03:40.463 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a781f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f26a013f080 0x7f26a013f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:40.463 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a781f640 10 monclient(hunting): picked mon.noname-a con 0x7f26a013f080 addr v2:172.21.15.7:3300/0 2023-11-25T05:03:40.464 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a781f640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a00aa970 con 0x7f26a00b2260 2023-11-25T05:03:40.464 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a781f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:40.464 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a781f640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a01421b0 con 0x7f26a013f080 2023-11-25T05:03:40.464 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a781f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:40.465 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a781f640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a00aac50 con 0x7f26a00a8a00 2023-11-25T05:03:40.465 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a781f640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:40.465 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a781f640 10 monclient(hunting): authenticate will time out at 1321.820680s 2023-11-25T05:03:40.465 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a4dba640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f26a00b2260 0x7f26a01425d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:59344/0 (socket says 172.21.15.90:59344) 2023-11-25T05:03:40.466 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a4dba640 1 -- 172.21.15.90:0/4292785963 learned_addr learned my addr 172.21.15.90:0/4292785963 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-11-25T05:03:40.466 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a5dbc640 1 --2- 172.21.15.90:0/4292785963 >> v2:172.21.15.7:3300/0 conn(0x7f26a013f080 0x7f26a013f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:40.466 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.454+0000 7f26a55bb640 1 --2- 172.21.15.90:0/4292785963 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f26a00a8a00 0x7f26a00a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:40.467 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.455+0000 7f26a5dbc640 10 monclient(hunting): get_auth_request con 0x7f26a013f080 auth_method 0 2023-11-25T05:03:40.469 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.455+0000 7f26a5dbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:40.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.455+0000 7f26a5dbc640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.455+0000 7f26a5dbc640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.455+0000 7f26a55bb640 10 monclient(hunting): get_auth_request con 0x7f26a00a8a00 auth_method 0 2023-11-25T05:03:40.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.455+0000 7f26a55bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:40.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.455+0000 7f26a55bb640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.455+0000 7f26a55bb640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.455+0000 7f26a5dbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:40.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.455+0000 7f26a5dbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:40.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.455+0000 7f26a5dbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:40.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.455+0000 7f26a55bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:40.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.455+0000 7f26a55bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:40.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.455+0000 7f26a55bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:40.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.457+0000 7f2697fff640 1 -- 172.21.15.90:0/4292785963 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608791108 0 0) 0x7f26a00aa970 con 0x7f26a00b2260 2023-11-25T05:03:40.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.457+0000 7f2697fff640 10 monclient(hunting): _negotiate 2023-11-25T05:03:40.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.457+0000 7f2697fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.457+0000 7f2697fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.457+0000 7f2697fff640 10 monclient(hunting): my global_id is 4135 2023-11-25T05:03:40.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.457+0000 7f2697fff640 1 -- 172.21.15.90:0/4292785963 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f268c003520 con 0x7f26a00b2260 2023-11-25T05:03:40.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.457+0000 7f26a5dbc640 10 monclient(hunting): handle_auth_done global_id 4137 payload 274 2023-11-25T05:03:40.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.457+0000 7f26a5dbc640 1 -- 172.21.15.90:0/4292785963 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f26a00a8a00 msgr2=0x7f26a00a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:40.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.457+0000 7f26a5dbc640 1 --2- 172.21.15.90:0/4292785963 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f26a00a8a00 0x7f26a00a8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.457+0000 7f26a5dbc640 1 -- 172.21.15.90:0/4292785963 >> v1:172.21.15.90:6789/0 conn(0x7f26a00b2260 legacy=0x7f26a01425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.457+0000 7f26a5dbc640 10 monclient: _finish_hunting 0 2023-11-25T05:03:40.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.457+0000 7f26a5dbc640 1 monclient: found mon.noname-a 2023-11-25T05:03:40.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.457+0000 7f26a5dbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:40.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.457+0000 7f26a5dbc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:03:40.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.457+0000 7f26a5dbc640 1 -- 172.21.15.90:0/4292785963 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a00aaf30 con 0x7f26a013f080 2023-11-25T05:03:40.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.459+0000 7f26a5dbc640 1 --2- 172.21.15.90:0/4292785963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a013f080 0x7f26a013f530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2698009d90 tx=0x7f269802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40817b2ae7afd8c server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.462+0000 7f2697fff640 1 -- 172.21.15.90:0/4292785963 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269802cb30 con 0x7f26a013f080 2023-11-25T05:03:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.462+0000 7f2697fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f2697fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:03:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f2697fff640 10 monclient: dump: 2023-11-25T05:03:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:40.479 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f2697fff640 10 monclient: _finish_auth 0 2023-11-25T05:03:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f2697fff640 10 monclient: _check_auth_tickets 2023-11-25T05:03:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f2697fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f2697fff640 1 -- 172.21.15.90:0/4292785963 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269802cc90 con 0x7f26a013f080 2023-11-25T05:03:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f26a781f640 5 monclient: authenticate success, global_id 4137 2023-11-25T05:03:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f26a781f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:03:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f2697fff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f26a781f640 10 monclient: get_monmap_and_config success 2023-11-25T05:03:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f26a781f640 4 set_mon_vals no callback set 2023-11-25T05:03:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f2697fff640 1 -- 172.21.15.90:0/4292785963 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2698034440 con 0x7f26a013f080 2023-11-25T05:03:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f2697fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:40.480 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f2697fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:03:40.480 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f2697fff640 10 monclient: dump: 2023-11-25T05:03:40.480 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:40.480 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:40.480 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.480 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.480 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:40.480 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.480 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:40.480 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:40.480 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:40.480 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.480 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f26a781f640 10 monclient: shutdown 2023-11-25T05:03:40.480 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f26a781f640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f26a781f640 1 -- 172.21.15.90:0/4292785963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a013f080 msgr2=0x7f26a013f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f26a781f640 1 --2- 172.21.15.90:0/4292785963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a013f080 0x7f26a013f530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2698009d90 tx=0x7f269802bd80 comp rx=0 tx=0).stop 2023-11-25T05:03:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f26a781f640 1 -- 172.21.15.90:0/4292785963 shutdown_connections 2023-11-25T05:03:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f26a781f640 1 --2- 172.21.15.90:0/4292785963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a013f080 0x7f26a013f530 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f26a781f640 1 --2- 172.21.15.90:0/4292785963 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f26a00a8a00 0x7f26a00a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f26a781f640 1 -- 172.21.15.90:0/4292785963 >> 172.21.15.90:0/4292785963 conn(0x7f26a0061a90 msgr2=0x7f26a00b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f26a781f640 1 -- 172.21.15.90:0/4292785963 shutdown_connections 2023-11-25T05:03:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.463+0000 7f26a781f640 1 -- 172.21.15.90:0/4292785963 wait complete. 2023-11-25T05:03:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.464+0000 7f26a781f640 10 monclient: build_initial_monmap 2023-11-25T05:03:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.464+0000 7f26a781f640 10 monclient: monmap: 2023-11-25T05:03:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:03:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2023-11-25T05:03:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2023-11-25T05:03:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:03:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:03:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:03:40.482 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.464+0000 7f26a781f640 1 librados: starting msgr at 2023-11-25T05:03:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.464+0000 7f26a781f640 1 librados: starting objecter 2023-11-25T05:03:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.464+0000 7f26a781f640 1 Processor -- start 2023-11-25T05:03:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.464+0000 7f26a781f640 1 -- start start 2023-11-25T05:03:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.464+0000 7f26a781f640 1 librados: setting wanted keys 2023-11-25T05:03:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.464+0000 7f26a781f640 1 librados: calling monclient init 2023-11-25T05:03:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.464+0000 7f26a781f640 10 monclient: init 2023-11-25T05:03:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-25T05:03:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26a00a8a00 0x7f26a013b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 10 monclient(hunting): picked mon.noname-b con 0x7f26a00a8a00 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a00b2260 0x7f26a01d6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 10 monclient(hunting): picked mon.noname-a con 0x7f26a00b2260 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f26a013f080 0x7f26a01339f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 10 monclient(hunting): picked mon.noname-c con 0x7f26a013f080 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a00b8150 con 0x7f26a00b2260 2023-11-25T05:03:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a013c1c0 con 0x7f26a00a8a00 2023-11-25T05:03:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a01d7960 con 0x7f26a013f080 2023-11-25T05:03:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a781f640 10 monclient(hunting): authenticate will time out at 1321.831913s 2023-11-25T05:03:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.465+0000 7f26a55bb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26a00a8a00 0x7f26a013b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a4dba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a00b2260 0x7f26a01d6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a5dbc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f26a013f080 0x7f26a01339f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a55bb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26a00a8a00 0x7f26a013b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40424/0 (socket says 172.21.15.90:40424) 2023-11-25T05:03:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a55bb640 1 -- 172.21.15.90:0/1282575661 learned_addr learned my addr 172.21.15.90:0/1282575661 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-11-25T05:03:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a55bb640 10 monclient(hunting): get_auth_request con 0x7f26a00a8a00 auth_method 0 2023-11-25T05:03:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a55bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a55bb640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a55bb640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a4dba640 10 monclient(hunting): get_auth_request con 0x7f26a00b2260 auth_method 0 2023-11-25T05:03:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a4dba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a4dba640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a4dba640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a5dbc640 10 monclient(hunting): get_auth_request con 0x7f26a013f080 auth_method 0 2023-11-25T05:03:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a5dbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a5dbc640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a5dbc640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a55bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a55bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a55bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a4dba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a4dba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.466+0000 7f26a4dba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.467+0000 7f26a5dbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.467+0000 7f26a5dbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.467+0000 7f26a5dbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.469+0000 7f26a55bb640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2023-11-25T05:03:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.469+0000 7f26a55bb640 1 -- 172.21.15.90:0/1282575661 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f26a013f080 msgr2=0x7f26a01339f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.469+0000 7f26a55bb640 1 --2- 172.21.15.90:0/1282575661 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f26a013f080 0x7f26a01339f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.469+0000 7f26a55bb640 1 -- 172.21.15.90:0/1282575661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a00b2260 msgr2=0x7f26a01d6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.469+0000 7f26a55bb640 1 --2- 172.21.15.90:0/1282575661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a00b2260 0x7f26a01d6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.469+0000 7f26a55bb640 10 monclient: _finish_hunting 0 2023-11-25T05:03:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.469+0000 7f26a55bb640 1 monclient: found mon.noname-b 2023-11-25T05:03:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.469+0000 7f26a55bb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.469+0000 7f26a55bb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.469+0000 7f26a55bb640 1 -- 172.21.15.90:0/1282575661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2698009a70 con 0x7f26a00a8a00 2023-11-25T05:03:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.471+0000 7f26a55bb640 1 --2- 172.21.15.90:0/1282575661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26a00a8a00 0x7f26a013b640 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f269000e930 tx=0x7f269000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.471+0000 7f2695ffb640 10 client.?.objecter ms_handle_connect 0x7f26a00a8a00 2023-11-25T05:03:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.471+0000 7f2695ffb640 10 client.?.objecter resend_mon_ops 2023-11-25T05:03:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f2695ffb640 1 -- 172.21.15.90:0/1282575661 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690009690 con 0x7f26a00a8a00 2023-11-25T05:03:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f2695ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f2695ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:03:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f2695ffb640 10 monclient: dump: 2023-11-25T05:03:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:40.488 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f2695ffb640 10 monclient: _finish_auth 0 2023-11-25T05:03:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f2695ffb640 10 monclient: _check_auth_tickets 2023-11-25T05:03:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f2695ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f2695ffb640 1 -- 172.21.15.90:0/1282575661 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269000ca00 con 0x7f26a00a8a00 2023-11-25T05:03:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f26a781f640 5 monclient: authenticate success, global_id 4138 2023-11-25T05:03:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f26a781f640 10 monclient: _renew_subs 2023-11-25T05:03:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f26a781f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f26a781f640 1 -- 172.21.15.90:0/1282575661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a01d7400 con 0x7f26a00a8a00 2023-11-25T05:03:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f2695ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f2695ffb640 1 -- 172.21.15.90:0/1282575661 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269000cdb0 con 0x7f26a00a8a00 2023-11-25T05:03:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f2695ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f2695ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:03:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f2695ffb640 10 monclient: dump: 2023-11-25T05:03:40.489 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:40.489 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:40.489 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.489 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.489 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:40.489 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.489 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:40.489 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:40.489 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:40.489 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.489 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f26977fe640 4 set_mon_vals no callback set 2023-11-25T05:03:40.489 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f26a781f640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:03:40.489 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f26a781f640 10 monclient: _renew_subs 2023-11-25T05:03:40.489 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f26a781f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.490 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f26a781f640 1 -- 172.21.15.90:0/1282575661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a0134110 con 0x7f26a00a8a00 2023-11-25T05:03:40.490 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.475+0000 7f26a781f640 1 librados: init done 2023-11-25T05:03:40.490 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.476+0000 7f26a781f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:40.490 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.476+0000 7f26a781f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:40.490 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.476+0000 7f26a781f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.490 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.476+0000 7f26a781f640 1 -- 172.21.15.90:0/1282575661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a0067ac0 con 0x7f26a00a8a00 2023-11-25T05:03:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.479+0000 7f2695ffb640 1 -- 172.21.15.90:0/1282575661 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2690005c90 con 0x7f26a00a8a00 2023-11-25T05:03:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.479+0000 7f2695ffb640 1 -- 172.21.15.90:0/1282575661 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f26900139f0 con 0x7f26a00a8a00 2023-11-25T05:03:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.479+0000 7f2695ffb640 10 client.4138.objecter ms_dispatch 0x7f26a0000b90 osd_map(7..7 src has 1..7) v4 2023-11-25T05:03:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.479+0000 7f2695ffb640 3 client.4138.objecter handle_osd_map got epochs [7,7] > 0 2023-11-25T05:03:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.479+0000 7f2695ffb640 3 client.4138.objecter handle_osd_map decoding full epoch 7 2023-11-25T05:03:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.479+0000 7f2695ffb640 20 client.4138.objecter dump_active .. 0 homeless 2023-11-25T05:03:40.578 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.577+0000 7f2695ffb640 1 -- 172.21.15.90:0/1282575661 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f2690021060 con 0x7f26a00a8a00 2023-11-25T05:03:40.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.577+0000 7f2695ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:40.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.577+0000 7f2695ffb640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:03:40.651 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.649+0000 7f26a781f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ebf452a0-1b17-488a-bbea-2e953c932ac7", "id": 5}] 2023-11-25T05:03:40.651 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.649+0000 7f26a781f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ebf452a0-1b17-488a-bbea-2e953c932ac7", "id": 5}] 2023-11-25T05:03:40.651 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.649+0000 7f26a781f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.651 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.649+0000 7f26a781f640 1 -- 172.21.15.90:0/1282575661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ebf452a0-1b17-488a-bbea-2e953c932ac7", "id": 5} v 0) v1 -- 0x7f26a01d7b80 con 0x7f26a00a8a00 2023-11-25T05:03:40.688 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:40.686+0000 7fdccda30040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-25T05:03:40.714 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.712+0000 7f2695ffb640 1 -- 172.21.15.90:0/1282575661 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ebf452a0-1b17-488a-bbea-2e953c932ac7", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2690017070 con 0x7f26a00a8a00 2023-11-25T05:03:40.714 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.712+0000 7f2695ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ebf452a0-1b17-488a-bbea-2e953c932ac7", "id": 5}] 2023-11-25T05:03:40.714 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.712+0000 7f2695ffb640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:03:40.714 INFO:teuthology.orchestra.run.smithi090.stdout:5 2023-11-25T05:03:40.716 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 10 librados: watch_flush enter 2023-11-25T05:03:40.716 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 10 librados: watch_flush exit 2023-11-25T05:03:40.716 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 20 client.4138.objecter shutdown clearing up homeless session... 2023-11-25T05:03:40.716 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 10 client.4138.objecter successfully canceled tick 2023-11-25T05:03:40.716 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 10 monclient: shutdown 2023-11-25T05:03:40.716 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:40.716 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 1 -- 172.21.15.90:0/1282575661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26a00a8a00 msgr2=0x7f26a013b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.716 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 1 --2- 172.21.15.90:0/1282575661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26a00a8a00 0x7f26a013b640 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f269000e930 tx=0x7f269000edf0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.716 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 1 -- 172.21.15.90:0/1282575661 shutdown_connections 2023-11-25T05:03:40.716 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 1 --2- 172.21.15.90:0/1282575661 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f26a013f080 0x7f26a01339f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.716 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 1 --2- 172.21.15.90:0/1282575661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a00b2260 0x7f26a01d6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.716 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 1 --2- 172.21.15.90:0/1282575661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26a00a8a00 0x7f26a013b640 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.716 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 1 -- 172.21.15.90:0/1282575661 >> 172.21.15.90:0/1282575661 conn(0x7f26a0061a90 msgr2=0x7f26a0134ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:40.717 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 1 -- 172.21.15.90:0/1282575661 shutdown_connections 2023-11-25T05:03:40.717 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 1 -- 172.21.15.90:0/1282575661 wait complete. 2023-11-25T05:03:40.717 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.713+0000 7f26a781f640 1 librados: shutdown 2023-11-25T05:03:40.725 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 5aaf35c3-5c06-45e5-b788-8d3207c47cd7 6 2023-11-25T05:03:40.819 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:40.817+0000 7f23a2432040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-25T05:03:40.829 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.827+0000 7ff896e90640 10 monclient: get_monmap_and_config 2023-11-25T05:03:40.829 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.827+0000 7ff896e90640 10 monclient: build_initial_monmap 2023-11-25T05:03:40.829 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.827+0000 7ff896e90640 10 monclient: monmap: 2023-11-25T05:03:40.829 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:40.829 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:40.829 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:03:40.828607+0000 2023-11-25T05:03:40.829 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:03:40.828607+0000 2023-11-25T05:03:40.829 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:03:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:03:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:03:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:03:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:03:40.830 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.831 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.828+0000 7ff896e90640 1 Processor -- start 2023-11-25T05:03:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.828+0000 7ff896e90640 1 -- start start 2023-11-25T05:03:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.828+0000 7ff896e90640 10 monclient: init 2023-11-25T05:03:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.828+0000 7ff896e90640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.828+0000 7ff896e90640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-11-25T05:03:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.828+0000 7ff896e90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8900a8a00 addr v1:172.21.15.7:6789/0 2023-11-25T05:03:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.828+0000 7ff896e90640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8900b2260 addr v1:172.21.15.90:6789/0 2023-11-25T05:03:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.828+0000 7ff896e90640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff8900a8de0 0x7ff89013f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.829+0000 7ff896e90640 10 monclient(hunting): picked mon.noname-c con 0x7ff8900a8de0 addr v2:172.21.15.90:3300/0 2023-11-25T05:03:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.829+0000 7ff896e90640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8900aa970 con 0x7ff8900a8a00 2023-11-25T05:03:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.829+0000 7ff896e90640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8900aac50 con 0x7ff8900b2260 2023-11-25T05:03:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.829+0000 7ff896e90640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.829+0000 7ff896e90640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8901421b0 con 0x7ff8900a8de0 2023-11-25T05:03:40.833 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.829+0000 7ff896e90640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:40.833 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.829+0000 7ff896e90640 10 monclient(hunting): authenticate will time out at 1322.195575s 2023-11-25T05:03:40.833 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.829+0000 7ff894c2c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff8900a8a00 0x7ff8901425d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.90:50862/0 (socket says 172.21.15.90:50862) 2023-11-25T05:03:40.833 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.829+0000 7ff894c2c640 1 -- 172.21.15.90:0/2105935183 learned_addr learned my addr 172.21.15.90:0/2105935183 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-11-25T05:03:40.833 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.829+0000 7ff89542d640 1 --2- 172.21.15.90:0/2105935183 >> v2:172.21.15.90:3300/0 conn(0x7ff8900a8de0 0x7ff89013f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:40.833 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.830+0000 7ff89542d640 10 monclient(hunting): get_auth_request con 0x7ff8900a8de0 auth_method 0 2023-11-25T05:03:40.833 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.830+0000 7ff89542d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:40.833 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.830+0000 7ff89542d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.833 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.830+0000 7ff89542d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.831+0000 7ff89542d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.831+0000 7ff89542d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.831+0000 7ff89542d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.831+0000 7ff88f7fe640 1 -- 172.21.15.90:0/2105935183 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891076050 0 0) 0x7ff8900aa970 con 0x7ff8900a8a00 2023-11-25T05:03:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.832+0000 7ff88f7fe640 10 monclient(hunting): _negotiate 2023-11-25T05:03:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.832+0000 7ff88f7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.832+0000 7ff88f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.832+0000 7ff88f7fe640 10 monclient(hunting): my global_id is 4143 2023-11-25T05:03:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.832+0000 7ff88f7fe640 1 -- 172.21.15.90:0/2105935183 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff878003520 con 0x7ff8900a8a00 2023-11-25T05:03:40.835 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.833+0000 7ff89542d640 10 monclient(hunting): handle_auth_done global_id 4141 payload 274 2023-11-25T05:03:40.835 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.833+0000 7ff89542d640 1 -- 172.21.15.90:0/2105935183 >> v1:172.21.15.90:6789/0 conn(0x7ff8900b2260 legacy=0x7ff89013f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.835 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.833+0000 7ff89542d640 1 -- 172.21.15.90:0/2105935183 >> v1:172.21.15.7:6789/0 conn(0x7ff8900a8a00 legacy=0x7ff8901425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.835 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.833+0000 7ff89542d640 10 monclient: _finish_hunting 0 2023-11-25T05:03:40.835 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.833+0000 7ff89542d640 1 monclient: found mon.noname-c 2023-11-25T05:03:40.835 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.833+0000 7ff89542d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:40.835 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.833+0000 7ff89542d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.836 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.833+0000 7ff89542d640 1 -- 172.21.15.90:0/2105935183 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8900aaf30 con 0x7ff8900a8de0 2023-11-25T05:03:40.838 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.837+0000 7ff89542d640 1 --2- 172.21.15.90:0/2105935183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8900a8de0 0x7ff89013f790 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff888009d90 tx=0x7ff88802be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=312d1890ca9c02a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:40.843 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff88f7fe640 1 -- 172.21.15.90:0/2105935183 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88802cbd0 con 0x7ff8900a8de0 2023-11-25T05:03:40.843 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff88f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:40.843 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff88f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:03:40.843 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff88f7fe640 10 monclient: dump: 2023-11-25T05:03:40.843 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:40.843 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:40.844 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff88f7fe640 10 monclient: _finish_auth 0 2023-11-25T05:03:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff88f7fe640 10 monclient: _check_auth_tickets 2023-11-25T05:03:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff88f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff88f7fe640 1 -- 172.21.15.90:0/2105935183 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88802cd30 con 0x7ff8900a8de0 2023-11-25T05:03:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff896e90640 5 monclient: authenticate success, global_id 4141 2023-11-25T05:03:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff896e90640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:03:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff88f7fe640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff896e90640 10 monclient: get_monmap_and_config success 2023-11-25T05:03:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff896e90640 4 set_mon_vals no callback set 2023-11-25T05:03:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff88f7fe640 1 -- 172.21.15.90:0/2105935183 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88802cea0 con 0x7ff8900a8de0 2023-11-25T05:03:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff88f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff88f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:03:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.841+0000 7ff88f7fe640 10 monclient: dump: 2023-11-25T05:03:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:40.846 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.849 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.842+0000 7ff896e90640 10 monclient: shutdown 2023-11-25T05:03:40.849 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.842+0000 7ff896e90640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:40.849 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.842+0000 7ff896e90640 1 -- 172.21.15.90:0/2105935183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8900a8de0 msgr2=0x7ff89013f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.842+0000 7ff896e90640 1 --2- 172.21.15.90:0/2105935183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8900a8de0 0x7ff89013f790 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff888009d90 tx=0x7ff88802be20 comp rx=0 tx=0).stop 2023-11-25T05:03:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.842+0000 7ff896e90640 1 -- 172.21.15.90:0/2105935183 shutdown_connections 2023-11-25T05:03:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.842+0000 7ff896e90640 1 --2- 172.21.15.90:0/2105935183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8900a8de0 0x7ff89013f790 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.842+0000 7ff896e90640 1 -- 172.21.15.90:0/2105935183 >> 172.21.15.90:0/2105935183 conn(0x7ff890061a90 msgr2=0x7ff8900b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.842+0000 7ff896e90640 1 -- 172.21.15.90:0/2105935183 shutdown_connections 2023-11-25T05:03:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.842+0000 7ff896e90640 1 -- 172.21.15.90:0/2105935183 wait complete. 2023-11-25T05:03:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.843+0000 7ff896e90640 10 monclient: build_initial_monmap 2023-11-25T05:03:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.843+0000 7ff896e90640 10 monclient: monmap: 2023-11-25T05:03:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:03:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2023-11-25T05:03:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2023-11-25T05:03:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:03:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:03:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:03:40.851 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.843+0000 7ff896e90640 1 librados: starting msgr at 2023-11-25T05:03:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.843+0000 7ff896e90640 1 librados: starting objecter 2023-11-25T05:03:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 1 Processor -- start 2023-11-25T05:03:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 1 -- start start 2023-11-25T05:03:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 1 librados: setting wanted keys 2023-11-25T05:03:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 1 librados: calling monclient init 2023-11-25T05:03:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 10 monclient: init 2023-11-25T05:03:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:03:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8900a8a00 0x7ff8901348e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 10 monclient(hunting): picked mon.noname-c con 0x7ff8900a8a00 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8900a8de0 0x7ff8901364d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 10 monclient(hunting): picked mon.noname-b con 0x7ff8900a8de0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8900b2260 0x7ff8901dad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 10 monclient(hunting): picked mon.noname-a con 0x7ff8900b2260 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff890146180 con 0x7ff8900b2260 2023-11-25T05:03:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff890135af0 con 0x7ff8900a8de0 2023-11-25T05:03:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff890135dd0 con 0x7ff8900a8a00 2023-11-25T05:03:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.844+0000 7ff896e90640 10 monclient(hunting): authenticate will time out at 1322.211370s 2023-11-25T05:03:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.845+0000 7ff88ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8900a8de0 0x7ff8901364d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.845+0000 7ff89542d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8900b2260 0x7ff8901dad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.845+0000 7ff894c2c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8900a8a00 0x7ff8901348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.845+0000 7ff88ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8900a8de0 0x7ff8901364d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40444/0 (socket says 172.21.15.90:40444) 2023-11-25T05:03:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.845+0000 7ff88ffff640 1 -- 172.21.15.90:0/2589339780 learned_addr learned my addr 172.21.15.90:0/2589339780 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-11-25T05:03:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.845+0000 7ff88ffff640 10 monclient(hunting): get_auth_request con 0x7ff8900a8de0 auth_method 0 2023-11-25T05:03:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.845+0000 7ff88ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.845+0000 7ff88ffff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.845+0000 7ff88ffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.845+0000 7ff89542d640 10 monclient(hunting): get_auth_request con 0x7ff8900b2260 auth_method 0 2023-11-25T05:03:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.845+0000 7ff89542d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.845+0000 7ff89542d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.845+0000 7ff89542d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.846+0000 7ff894c2c640 10 monclient(hunting): get_auth_request con 0x7ff8900a8a00 auth_method 0 2023-11-25T05:03:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.846+0000 7ff894c2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.846+0000 7ff894c2c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.846+0000 7ff894c2c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.846+0000 7ff88ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.846+0000 7ff88ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.846+0000 7ff88ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.846+0000 7ff89542d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.846+0000 7ff89542d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.846+0000 7ff89542d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.846+0000 7ff894c2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.846+0000 7ff894c2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.846+0000 7ff894c2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.848+0000 7ff88ffff640 10 monclient(hunting): handle_auth_done global_id 4147 payload 824 2023-11-25T05:03:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.848+0000 7ff88ffff640 1 -- 172.21.15.90:0/2589339780 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8900a8a00 msgr2=0x7ff8901348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.848+0000 7ff88ffff640 1 --2- 172.21.15.90:0/2589339780 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8900a8a00 0x7ff8901348e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.848+0000 7ff88ffff640 1 -- 172.21.15.90:0/2589339780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8900b2260 msgr2=0x7ff8901dad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.848+0000 7ff88ffff640 1 --2- 172.21.15.90:0/2589339780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8900b2260 0x7ff8901dad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.848+0000 7ff88ffff640 10 monclient: _finish_hunting 0 2023-11-25T05:03:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.848+0000 7ff88ffff640 1 monclient: found mon.noname-b 2023-11-25T05:03:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.848+0000 7ff88ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.848+0000 7ff88ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.848+0000 7ff88ffff640 1 -- 172.21.15.90:0/2589339780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff888009a70 con 0x7ff8900a8de0 2023-11-25T05:03:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.851+0000 7ff88ffff640 1 --2- 172.21.15.90:0/2589339780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8900a8de0 0x7ff8901364d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff88400c440 tx=0x7ff88400c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.851+0000 7ff88d7fa640 10 client.?.objecter ms_handle_connect 0x7ff8900a8de0 2023-11-25T05:03:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.851+0000 7ff88d7fa640 10 client.?.objecter resend_mon_ops 2023-11-25T05:03:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff88d7fa640 1 -- 172.21.15.90:0/2589339780 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8840052d0 con 0x7ff8900a8de0 2023-11-25T05:03:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff88d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff88d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:03:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff88d7fa640 10 monclient: dump: 2023-11-25T05:03:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:40.859 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff88d7fa640 10 monclient: _finish_auth 0 2023-11-25T05:03:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff88d7fa640 10 monclient: _check_auth_tickets 2023-11-25T05:03:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff88d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff88d7fa640 1 -- 172.21.15.90:0/2589339780 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff884005430 con 0x7ff8900a8de0 2023-11-25T05:03:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff88d7fa640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff88d7fa640 1 -- 172.21.15.90:0/2589339780 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff884007260 con 0x7ff8900a8de0 2023-11-25T05:03:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff88d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff88d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:03:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff88d7fa640 10 monclient: dump: 2023-11-25T05:03:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:40.860 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff896e90640 5 monclient: authenticate success, global_id 4147 2023-11-25T05:03:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff896e90640 10 monclient: _renew_subs 2023-11-25T05:03:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff896e90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff896e90640 1 -- 172.21.15.90:0/2589339780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8901360b0 con 0x7ff8900a8de0 2023-11-25T05:03:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff88effd640 4 set_mon_vals no callback set 2023-11-25T05:03:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff896e90640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:03:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff896e90640 10 monclient: _renew_subs 2023-11-25T05:03:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff896e90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff896e90640 1 -- 172.21.15.90:0/2589339780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff890145800 con 0x7ff8900a8de0 2023-11-25T05:03:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.854+0000 7ff896e90640 1 librados: init done 2023-11-25T05:03:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.855+0000 7ff896e90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.855+0000 7ff896e90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.855+0000 7ff896e90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.855+0000 7ff896e90640 1 -- 172.21.15.90:0/2589339780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff890067ac0 con 0x7ff8900a8de0 2023-11-25T05:03:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.858+0000 7ff88d7fa640 1 -- 172.21.15.90:0/2589339780 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff8840067d0 con 0x7ff8900a8de0 2023-11-25T05:03:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.858+0000 7ff88d7fa640 1 -- 172.21.15.90:0/2589339780 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7ff884011590 con 0x7ff8900a8de0 2023-11-25T05:03:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.858+0000 7ff88d7fa640 10 client.4147.objecter ms_dispatch 0x7ff890000b90 osd_map(8..8 src has 1..8) v4 2023-11-25T05:03:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.858+0000 7ff88d7fa640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2023-11-25T05:03:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.858+0000 7ff88d7fa640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2023-11-25T05:03:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.858+0000 7ff88d7fa640 20 client.4147.objecter dump_active .. 0 homeless 2023-11-25T05:03:40.959 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.958+0000 7ff88d7fa640 1 -- 172.21.15.90:0/2589339780 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff884020060 con 0x7ff8900a8de0 2023-11-25T05:03:40.960 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.958+0000 7ff88d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:40.960 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:40.958+0000 7ff88d7fa640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:03:40.979 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:40.977+0000 7fdccda30040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-25T05:03:41.031 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.030+0000 7ff896e90640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5aaf35c3-5c06-45e5-b788-8d3207c47cd7", "id": 6}] 2023-11-25T05:03:41.031 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.030+0000 7ff896e90640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5aaf35c3-5c06-45e5-b788-8d3207c47cd7", "id": 6}] 2023-11-25T05:03:41.031 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.030+0000 7ff896e90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:41.032 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:41.030+0000 7fdccda30040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-25T05:03:41.032 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.030+0000 7ff896e90640 1 -- 172.21.15.90:0/2589339780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5aaf35c3-5c06-45e5-b788-8d3207c47cd7", "id": 6} v 0) v1 -- 0x7ff8901462f0 con 0x7ff8900a8de0 2023-11-25T05:03:41.092 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.091+0000 7ff88d7fa640 1 -- 172.21.15.90:0/2589339780 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5aaf35c3-5c06-45e5-b788-8d3207c47cd7", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff884006a80 con 0x7ff8900a8de0 2023-11-25T05:03:41.093 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.091+0000 7ff88d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5aaf35c3-5c06-45e5-b788-8d3207c47cd7", "id": 6}] 2023-11-25T05:03:41.093 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.091+0000 7ff88d7fa640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:03:41.093 INFO:teuthology.orchestra.run.smithi090.stdout:6 2023-11-25T05:03:41.095 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 10 librados: watch_flush enter 2023-11-25T05:03:41.095 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 10 librados: watch_flush exit 2023-11-25T05:03:41.095 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 20 client.4147.objecter shutdown clearing up homeless session... 2023-11-25T05:03:41.095 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 10 client.4147.objecter successfully canceled tick 2023-11-25T05:03:41.095 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 10 monclient: shutdown 2023-11-25T05:03:41.095 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:41.096 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 1 -- 172.21.15.90:0/2589339780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8900a8de0 msgr2=0x7ff8901364d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:41.096 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 1 --2- 172.21.15.90:0/2589339780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8900a8de0 0x7ff8901364d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff88400c440 tx=0x7ff88400c900 comp rx=0 tx=0).stop 2023-11-25T05:03:41.096 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 1 -- 172.21.15.90:0/2589339780 shutdown_connections 2023-11-25T05:03:41.096 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 1 --2- 172.21.15.90:0/2589339780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8900b2260 0x7ff8901dad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:41.096 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 1 --2- 172.21.15.90:0/2589339780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8900a8de0 0x7ff8901364d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:41.096 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 1 --2- 172.21.15.90:0/2589339780 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8900a8a00 0x7ff8901348e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:41.097 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 1 -- 172.21.15.90:0/2589339780 >> 172.21.15.90:0/2589339780 conn(0x7ff890061a90 msgr2=0x7ff8900b6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:41.097 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 1 -- 172.21.15.90:0/2589339780 shutdown_connections 2023-11-25T05:03:41.097 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 1 -- 172.21.15.90:0/2589339780 wait complete. 2023-11-25T05:03:41.097 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.092+0000 7ff896e90640 1 librados: shutdown 2023-11-25T05:03:41.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new d50dd23a-f864-4593-ac00-4d8afed21dd0 7 2023-11-25T05:03:41.108 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:41.107+0000 7f23a2432040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-25T05:03:41.137 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:41.136+0000 7fdccda30040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-25T05:03:41.162 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:41.161+0000 7f23a2432040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-25T05:03:41.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.200+0000 7f46873c0640 10 monclient: get_monmap_and_config 2023-11-25T05:03:41.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.200+0000 7f46873c0640 10 monclient: build_initial_monmap 2023-11-25T05:03:41.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.200+0000 7f46873c0640 10 monclient: monmap: 2023-11-25T05:03:41.202 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:41.202 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:41.202 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:03:41.201849+0000 2023-11-25T05:03:41.203 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:03:41.201849+0000 2023-11-25T05:03:41.203 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:41.203 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:41.203 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:03:41.203 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:03:41.203 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:03:41.203 INFO:teuthology.orchestra.run.smithi090.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:03:41.203 INFO:teuthology.orchestra.run.smithi090.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:03:41.203 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:41.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.201+0000 7f46873c0640 1 Processor -- start 2023-11-25T05:03:41.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.201+0000 7f46873c0640 1 -- start start 2023-11-25T05:03:41.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.201+0000 7f46873c0640 10 monclient: init 2023-11-25T05:03:41.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.202+0000 7f46873c0640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:41.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.202+0000 7f46873c0640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-11-25T05:03:41.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.202+0000 7f46873c0640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f468006a770 0x7f468013fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:41.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.202+0000 7f46873c0640 10 monclient(hunting): picked mon.noname-a con 0x7f468006a770 addr v2:172.21.15.7:3300/0 2023-11-25T05:03:41.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.202+0000 7f46873c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46800afaf0 addr v1:172.21.15.90:6789/0 2023-11-25T05:03:41.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.202+0000 7f46873c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f468013d3f0 addr v1:172.21.15.7:6789/0 2023-11-25T05:03:41.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.202+0000 7f46873c0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468013d860 con 0x7f468013d3f0 2023-11-25T05:03:41.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.202+0000 7f46873c0640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46800a86f0 con 0x7f46800afaf0 2023-11-25T05:03:41.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.202+0000 7f46873c0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:41.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.202+0000 7f46873c0640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f468006ab50 con 0x7f468006a770 2023-11-25T05:03:41.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.202+0000 7f46873c0640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:41.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.202+0000 7f46873c0640 10 monclient(hunting): authenticate will time out at 1322.568917s 2023-11-25T05:03:41.206 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.202+0000 7f468495b640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f46800afaf0 0x7f468013cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:59376/0 (socket says 172.21.15.90:59376) 2023-11-25T05:03:41.206 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.202+0000 7f468495b640 1 -- 172.21.15.90:0/883226779 learned_addr learned my addr 172.21.15.90:0/883226779 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-11-25T05:03:41.206 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.205+0000 7f4677fff640 1 -- 172.21.15.90:0/883226779 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173180719 0 0) 0x7f46800a86f0 con 0x7f46800afaf0 2023-11-25T05:03:41.206 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.205+0000 7f4677fff640 10 monclient(hunting): _negotiate 2023-11-25T05:03:41.206 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.205+0000 7f4677fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:41.206 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.205+0000 7f4677fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:41.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.205+0000 7f4677fff640 10 monclient(hunting): my global_id is 4150 2023-11-25T05:03:41.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.205+0000 7f4677fff640 1 -- 172.21.15.90:0/883226779 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f466c003520 con 0x7f46800afaf0 2023-11-25T05:03:41.212 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.209+0000 7f4677fff640 1 -- 172.21.15.90:0/883226779 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1841382785 0 0) 0x7f466c003520 con 0x7f46800afaf0 2023-11-25T05:03:41.212 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 1 -- 172.21.15.90:0/883226779 >> v2:172.21.15.7:3300/0 conn(0x7f468006a770 msgr2=0x7f468013fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:41.212 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 1 --2- 172.21.15.90:0/883226779 >> v2:172.21.15.7:3300/0 conn(0x7f468006a770 0x7f468013fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:41.212 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 1 -- 172.21.15.90:0/883226779 >> v1:172.21.15.7:6789/0 conn(0x7f468013d3f0 legacy=0x7f468014a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient: _finish_hunting 0 2023-11-25T05:03:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 1 monclient: found mon.noname-c-legacy 2023-11-25T05:03:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.90:6789/0 2023-11-25T05:03:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 1 -- 172.21.15.90:0/883226779 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46800a89d0 con 0x7f46800afaf0 2023-11-25T05:03:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient: _finish_auth 0 2023-11-25T05:03:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient: _check_auth_tickets 2023-11-25T05:03:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f46873c0640 5 monclient: authenticate success, global_id 4150 2023-11-25T05:03:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 1 -- 172.21.15.90:0/883226779 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4243585761 0 0) 0x7f4670001670 con 0x7f46800afaf0 2023-11-25T05:03:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f46873c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:03:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-11-25T05:03:41.214 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient: dump: 2023-11-25T05:03:41.214 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:41.214 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:41.214 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:41.214 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:41.214 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:41.214 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:41.214 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:41.214 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:41.214 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:41.214 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:41.214 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] but i'm connected to v1:172.21.15.90:6789/0, reconnecting 2023-11-25T05:03:41.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:41.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 1 -- 172.21.15.90:0/883226779 >> v1:172.21.15.90:6789/0 conn(0x7f46800afaf0 legacy=0x7f468013cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:41.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-25T05:03:41.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 1 --2- 172.21.15.90:0/883226779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f466c0039c0 0x7f466c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:41.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient(hunting): picked mon.b con 0x7f466c0039c0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:41.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 1 --2- 172.21.15.90:0/883226779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f466c0047c0 0x7f466c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:41.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient(hunting): picked mon.a con 0x7f466c0047c0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:41.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 1 --2- 172.21.15.90:0/883226779 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f466c005330 0x7f466c00b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:41.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient(hunting): picked mon.c con 0x7f466c005330 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:41.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:41.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 1 -- 172.21.15.90:0/883226779 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468006ab50 con 0x7f466c0047c0 2023-11-25T05:03:41.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:41.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 1 -- 172.21.15.90:0/883226779 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c00c280 con 0x7f466c0039c0 2023-11-25T05:03:41.216 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:41.216 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 1 -- 172.21.15.90:0/883226779 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466c00c440 con 0x7f466c005330 2023-11-25T05:03:41.216 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:41.216 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f4677fff640 10 monclient(hunting): _finish_auth 0 2023-11-25T05:03:41.216 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f46873c0640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-11-25T05:03:41.216 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.210+0000 7f468595d640 1 --2- 172.21.15.90:0/883226779 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f466c005330 0x7f466c00b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:41.216 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.211+0000 7f468595d640 10 monclient(hunting): get_auth_request con 0x7f466c005330 auth_method 0 2023-11-25T05:03:41.216 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.211+0000 7f468595d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:41.216 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.211+0000 7f468595d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:41.216 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.211+0000 7f468595d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:41.218 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.211+0000 7f468595d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:41.218 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.211+0000 7f468595d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:41.218 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.211+0000 7f468595d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-11-25T05:03:41.218 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468515c640 1 --2- 172.21.15.90:0/883226779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f466c0039c0 0x7f466c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:41.218 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468515c640 10 monclient(hunting): get_auth_request con 0x7f466c0039c0 auth_method 0 2023-11-25T05:03:41.218 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468515c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:41.218 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468515c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:41.218 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468515c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:41.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468515c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:41.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468515c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:41.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468515c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-11-25T05:03:41.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468595d640 10 monclient(hunting): handle_auth_done global_id 4150 payload 293 2023-11-25T05:03:41.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468595d640 1 -- 172.21.15.90:0/883226779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f466c0039c0 msgr2=0x7f466c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:41.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468595d640 1 --2- 172.21.15.90:0/883226779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f466c0039c0 0x7f466c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:41.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468595d640 1 -- 172.21.15.90:0/883226779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f466c0047c0 msgr2=0x7f466c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:41.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468595d640 1 --2- 172.21.15.90:0/883226779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f466c0047c0 0x7f466c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:41.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468595d640 10 monclient: _finish_hunting 0 2023-11-25T05:03:41.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468595d640 1 monclient: found mon.c 2023-11-25T05:03:41.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468595d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:41.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468595d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:03:41.220 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468595d640 1 -- 172.21.15.90:0/883226779 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c003520 con 0x7f466c005330 2023-11-25T05:03:41.220 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468595d640 10 monclient: _finish_auth 0 2023-11-25T05:03:41.220 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468595d640 10 monclient: _check_auth_tickets 2023-11-25T05:03:41.220 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.213+0000 7f468595d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:41.220 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.214+0000 7f468595d640 1 -- 172.21.15.90:0/883226779 reap_dead start 2023-11-25T05:03:41.220 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.216+0000 7f468595d640 1 --2- 172.21.15.90:0/883226779 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f466c005330 0x7f466c00b9a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4678001d50 tx=0x7f466c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=338f84aad860929e server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:41.227 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f4677fff640 1 -- 172.21.15.90:0/883226779 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4678034070 con 0x7f466c005330 2023-11-25T05:03:41.227 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f4677fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:41.227 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f4677fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:03:41.228 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f4677fff640 10 monclient: dump: 2023-11-25T05:03:41.228 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:41.228 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:41.228 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:41.228 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:41.228 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:41.229 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:41.229 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:41.229 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:41.229 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:41.229 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:41.229 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f46873c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:03:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f4677fff640 1 -- 172.21.15.90:0/883226779 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4678003990 con 0x7f466c005330 2023-11-25T05:03:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f4677fff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f46873c0640 10 monclient: get_monmap_and_config success 2023-11-25T05:03:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f46873c0640 4 set_mon_vals no callback set 2023-11-25T05:03:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f4677fff640 1 -- 172.21.15.90:0/883226779 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4678003b40 con 0x7f466c005330 2023-11-25T05:03:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f4677fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:41.231 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f4677fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:03:41.231 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f4677fff640 10 monclient: dump: 2023-11-25T05:03:41.231 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:41.231 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:41.231 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:41.231 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:41.232 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:41.232 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:41.232 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:41.232 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:41.232 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:41.232 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:41.233 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f46873c0640 10 monclient: shutdown 2023-11-25T05:03:41.233 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f46873c0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:41.233 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f46873c0640 1 -- 172.21.15.90:0/883226779 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f466c005330 msgr2=0x7f466c00b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:41.233 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f46873c0640 1 --2- 172.21.15.90:0/883226779 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f466c005330 0x7f466c00b9a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4678001d50 tx=0x7f466c0057f0 comp rx=0 tx=0).stop 2023-11-25T05:03:41.233 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f46873c0640 1 -- 172.21.15.90:0/883226779 shutdown_connections 2023-11-25T05:03:41.233 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f46873c0640 1 --2- 172.21.15.90:0/883226779 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f466c005330 0x7f466c00b9a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:41.234 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f46873c0640 1 -- 172.21.15.90:0/883226779 >> 172.21.15.90:0/883226779 conn(0x7f46800b4b10 msgr2=0x7f46800b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:41.234 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f46873c0640 1 -- 172.21.15.90:0/883226779 shutdown_connections 2023-11-25T05:03:41.234 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.221+0000 7f46873c0640 1 -- 172.21.15.90:0/883226779 wait complete. 2023-11-25T05:03:41.234 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.222+0000 7f46873c0640 10 monclient: build_initial_monmap 2023-11-25T05:03:41.234 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.222+0000 7f46873c0640 10 monclient: monmap: 2023-11-25T05:03:41.234 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2023-11-25T05:03:41.235 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:03:41.235 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2023-11-25T05:03:41.235 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2023-11-25T05:03:41.235 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:41.235 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:41.235 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:03:41.236 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:03:41.236 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:03:41.236 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:41.236 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.222+0000 7f46873c0640 1 librados: starting msgr at 2023-11-25T05:03:41.236 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.222+0000 7f46873c0640 1 librados: starting objecter 2023-11-25T05:03:41.236 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.222+0000 7f46873c0640 1 Processor -- start 2023-11-25T05:03:41.237 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.222+0000 7f46873c0640 1 -- start start 2023-11-25T05:03:41.237 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.222+0000 7f46873c0640 1 librados: setting wanted keys 2023-11-25T05:03:41.237 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.222+0000 7f46873c0640 1 librados: calling monclient init 2023-11-25T05:03:41.237 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.222+0000 7f46873c0640 10 monclient: init 2023-11-25T05:03:41.237 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:41.237 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-25T05:03:41.238 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f466c005330 0x7f46800ad840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:41.238 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 10 monclient(hunting): picked mon.noname-a con 0x7f466c005330 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:41.238 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4680137770 0x7f46801d7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:41.238 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 10 monclient(hunting): picked mon.noname-b con 0x7f4680137770 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:41.238 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46801daf60 0x7f468013f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:41.239 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 10 monclient(hunting): picked mon.noname-c con 0x7f46801daf60 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:41.239 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:41.239 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4680140c30 con 0x7f466c005330 2023-11-25T05:03:41.239 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:41.239 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46801daad0 con 0x7f4680137770 2023-11-25T05:03:41.239 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:41.240 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46801da760 con 0x7f46801daf60 2023-11-25T05:03:41.240 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:41.240 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.223+0000 7f46873c0640 10 monclient(hunting): authenticate will time out at 1322.589986s 2023-11-25T05:03:41.240 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.224+0000 7f468595d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46801daf60 0x7f468013f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:41.240 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.224+0000 7f468595d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46801daf60 0x7f468013f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.90:60948/0 (socket says 172.21.15.90:60948) 2023-11-25T05:03:41.240 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.224+0000 7f468595d640 1 -- 172.21.15.90:0/3020147480 learned_addr learned my addr 172.21.15.90:0/3020147480 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-11-25T05:03:41.240 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.224+0000 7f468595d640 10 monclient(hunting): get_auth_request con 0x7f46801daf60 auth_method 0 2023-11-25T05:03:41.241 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.224+0000 7f468595d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:41.241 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.224+0000 7f468595d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:41.241 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.224+0000 7f468595d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:41.241 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.225+0000 7f468595d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:41.241 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.225+0000 7f468595d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:41.241 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.225+0000 7f468595d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:41.244 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.226+0000 7f468495b640 1 --2- 172.21.15.90:0/3020147480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4680137770 0x7f46801d7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:41.244 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.227+0000 7f468515c640 1 --2- 172.21.15.90:0/3020147480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f466c005330 0x7f46800ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:41.245 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.228+0000 7f468515c640 10 monclient(hunting): get_auth_request con 0x7f466c005330 auth_method 0 2023-11-25T05:03:41.245 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.228+0000 7f468515c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:41.245 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.228+0000 7f468515c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:41.245 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.228+0000 7f468515c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:41.245 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.228+0000 7f468595d640 10 monclient(hunting): handle_auth_done global_id 4136 payload 824 2023-11-25T05:03:41.245 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.228+0000 7f468595d640 1 -- 172.21.15.90:0/3020147480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4680137770 msgr2=0x7f46801d7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:41.245 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.228+0000 7f468595d640 1 --2- 172.21.15.90:0/3020147480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4680137770 0x7f46801d7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:41.245 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.228+0000 7f468595d640 1 -- 172.21.15.90:0/3020147480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f466c005330 msgr2=0x7f46800ad840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:41.246 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.228+0000 7f468595d640 1 --2- 172.21.15.90:0/3020147480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f466c005330 0x7f46800ad840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:41.246 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.228+0000 7f468595d640 10 monclient: _finish_hunting 0 2023-11-25T05:03:41.246 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.228+0000 7f468595d640 1 monclient: found mon.noname-c 2023-11-25T05:03:41.246 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.228+0000 7f468595d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:41.246 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.228+0000 7f468595d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:03:41.246 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.228+0000 7f468595d640 1 -- 172.21.15.90:0/3020147480 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4680138210 con 0x7f46801daf60 2023-11-25T05:03:41.247 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.246+0000 7f468595d640 1 --2- 172.21.15.90:0/3020147480 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46801daf60 0x7f468013f9d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4678001e40 tx=0x7f4678001330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:41.247 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.246+0000 7f4675ffb640 10 client.?.objecter ms_handle_connect 0x7f46801daf60 2023-11-25T05:03:41.247 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.246+0000 7f4675ffb640 10 client.?.objecter resend_mon_ops 2023-11-25T05:03:41.258 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f4675ffb640 1 -- 172.21.15.90:0/3020147480 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4678034040 con 0x7f46801daf60 2023-11-25T05:03:41.258 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f4675ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:41.258 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f4675ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:03:41.258 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f4675ffb640 10 monclient: dump: 2023-11-25T05:03:41.258 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:41.258 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:41.258 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:41.259 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f4675ffb640 10 monclient: _finish_auth 0 2023-11-25T05:03:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f4675ffb640 10 monclient: _check_auth_tickets 2023-11-25T05:03:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f4675ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f4675ffb640 1 -- 172.21.15.90:0/3020147480 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4678006b30 con 0x7f46801daf60 2023-11-25T05:03:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f4675ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f4675ffb640 1 -- 172.21.15.90:0/3020147480 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4678005450 con 0x7f46801daf60 2023-11-25T05:03:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f4675ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f4675ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:03:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f4675ffb640 10 monclient: dump: 2023-11-25T05:03:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2023-11-25T05:03:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2023-11-25T05:03:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:41.260 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:03:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f46873c0640 5 monclient: authenticate success, global_id 4136 2023-11-25T05:03:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f46873c0640 10 monclient: _renew_subs 2023-11-25T05:03:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f46873c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:03:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f46777fe640 4 set_mon_vals no callback set 2023-11-25T05:03:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f46873c0640 1 -- 172.21.15.90:0/3020147480 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46801db3e0 con 0x7f46801daf60 2023-11-25T05:03:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f46873c0640 10 client.4136.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:03:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f46873c0640 10 monclient: _renew_subs 2023-11-25T05:03:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f46873c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:03:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f46873c0640 1 -- 172.21.15.90:0/3020147480 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46801400f0 con 0x7f46801daf60 2023-11-25T05:03:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.255+0000 7f46873c0640 1 librados: init done 2023-11-25T05:03:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.256+0000 7f46873c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.256+0000 7f46873c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.256+0000 7f46873c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:03:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.256+0000 7f46873c0640 1 -- 172.21.15.90:0/3020147480 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46801da920 con 0x7f46801daf60 2023-11-25T05:03:41.275 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-11-25T05:03:41.272+0000 7f23a2432040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-25T05:03:41.361 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.360+0000 7f4675ffb640 1 -- 172.21.15.90:0/3020147480 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4678004030 con 0x7f46801daf60 2023-11-25T05:03:41.371 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.365+0000 7f4675ffb640 1 -- 172.21.15.90:0/3020147480 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f4678044aa0 con 0x7f46801daf60 2023-11-25T05:03:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.365+0000 7f4675ffb640 10 client.4136.objecter ms_dispatch 0x7f4680000b90 osd_map(9..9 src has 1..9) v4 2023-11-25T05:03:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.365+0000 7f4675ffb640 3 client.4136.objecter handle_osd_map got epochs [9,9] > 0 2023-11-25T05:03:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.365+0000 7f4675ffb640 3 client.4136.objecter handle_osd_map decoding full epoch 9 2023-11-25T05:03:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.365+0000 7f4675ffb640 20 client.4136.objecter dump_active .. 0 homeless 2023-11-25T05:03:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.365+0000 7f4675ffb640 1 -- 172.21.15.90:0/3020147480 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f4678001920 con 0x7f46801daf60 2023-11-25T05:03:41.372 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.366+0000 7f4675ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:41.373 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.366+0000 7f4675ffb640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:03:41.442 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.438+0000 7f46873c0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d50dd23a-f864-4593-ac00-4d8afed21dd0", "id": 7}] 2023-11-25T05:03:41.442 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.438+0000 7f46873c0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d50dd23a-f864-4593-ac00-4d8afed21dd0", "id": 7}] 2023-11-25T05:03:41.442 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.438+0000 7f46873c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:03:41.442 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.438+0000 7f46873c0640 1 -- 172.21.15.90:0/3020147480 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d50dd23a-f864-4593-ac00-4d8afed21dd0", "id": 7} v 0) v1 -- 0x7f46800b2ad0 con 0x7f46801daf60 2023-11-25T05:03:41.496 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.495+0000 7f4675ffb640 1 -- 172.21.15.90:0/3020147480 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mgrmap(e 2) v1 ==== 44567+0+0 (secure 0 0 0) 0x7f4678044440 con 0x7f46801daf60 2023-11-25T05:03:41.826 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.825+0000 7f4675ffb640 1 -- 172.21.15.90:0/3020147480 <== mon.2 v2:172.21.15.7:3301/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d50dd23a-f864-4593-ac00-4d8afed21dd0", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f46780072d0 con 0x7f46801daf60 2023-11-25T05:03:41.827 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.825+0000 7f4675ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d50dd23a-f864-4593-ac00-4d8afed21dd0", "id": 7}] 2023-11-25T05:03:41.827 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.825+0000 7f4675ffb640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:03:41.827 INFO:teuthology.orchestra.run.smithi090.stdout:7 2023-11-25T05:03:41.828 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.825+0000 7f46873c0640 10 librados: watch_flush enter 2023-11-25T05:03:41.828 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.825+0000 7f46873c0640 10 librados: watch_flush exit 2023-11-25T05:03:41.828 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.825+0000 7f46873c0640 20 client.4136.objecter shutdown clearing up homeless session... 2023-11-25T05:03:41.828 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.825+0000 7f46873c0640 10 client.4136.objecter successfully canceled tick 2023-11-25T05:03:41.829 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.826+0000 7f46873c0640 10 monclient: shutdown 2023-11-25T05:03:41.829 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.826+0000 7f46873c0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:41.829 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.826+0000 7f46873c0640 1 -- 172.21.15.90:0/3020147480 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46801daf60 msgr2=0x7f468013f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:41.829 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.826+0000 7f46873c0640 1 --2- 172.21.15.90:0/3020147480 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46801daf60 0x7f468013f9d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4678001e40 tx=0x7f4678001330 comp rx=0 tx=0).stop 2023-11-25T05:03:41.829 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.826+0000 7f46873c0640 1 -- 172.21.15.90:0/3020147480 shutdown_connections 2023-11-25T05:03:41.829 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.826+0000 7f46873c0640 1 --2- 172.21.15.90:0/3020147480 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46801daf60 0x7f468013f9d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:41.829 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.826+0000 7f46873c0640 1 --2- 172.21.15.90:0/3020147480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4680137770 0x7f46801d7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:41.829 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.826+0000 7f46873c0640 1 --2- 172.21.15.90:0/3020147480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f466c005330 0x7f46800ad840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:41.829 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.826+0000 7f46873c0640 1 -- 172.21.15.90:0/3020147480 >> 172.21.15.90:0/3020147480 conn(0x7f46800b4b10 msgr2=0x7f4680073c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:41.829 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.826+0000 7f46873c0640 1 -- 172.21.15.90:0/3020147480 shutdown_connections 2023-11-25T05:03:41.829 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.826+0000 7f46873c0640 1 -- 172.21.15.90:0/3020147480 wait complete. 2023-11-25T05:03:41.829 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-25T05:03:41.826+0000 7f46873c0640 1 librados: shutdown 2023-11-25T05:03:41.837 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-25T05:03:41.837 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-25T05:03:41.838 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-11-25T05:03:41.840 INFO:tasks.ceph.osd.0:Started 2023-11-25T05:03:41.840 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-25T05:03:41.841 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-25T05:03:41.841 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-11-25T05:03:41.843 INFO:tasks.ceph.osd.1:Started 2023-11-25T05:03:41.843 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-25T05:03:41.843 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-25T05:03:41.843 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-11-25T05:03:41.845 INFO:tasks.ceph.osd.2:Started 2023-11-25T05:03:41.845 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-25T05:03:41.846 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-25T05:03:41.846 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-11-25T05:03:41.848 INFO:tasks.ceph.osd.3:Started 2023-11-25T05:03:41.848 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-25T05:03:41.848 INFO:tasks.ceph.osd.4:Restarting daemon 2023-11-25T05:03:41.848 DEBUG:teuthology.orchestra.run.smithi090:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-11-25T05:03:41.880 INFO:tasks.ceph.osd.4:Started 2023-11-25T05:03:41.880 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-25T05:03:41.880 INFO:tasks.ceph.osd.5:Restarting daemon 2023-11-25T05:03:41.880 DEBUG:teuthology.orchestra.run.smithi090:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-11-25T05:03:41.882 INFO:tasks.ceph.osd.5:Started 2023-11-25T05:03:41.882 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-25T05:03:41.882 INFO:tasks.ceph.osd.6:Restarting daemon 2023-11-25T05:03:41.882 DEBUG:teuthology.orchestra.run.smithi090:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-11-25T05:03:41.884 INFO:tasks.ceph.osd.6:Started 2023-11-25T05:03:41.884 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-25T05:03:41.884 INFO:tasks.ceph.osd.7:Restarting daemon 2023-11-25T05:03:41.885 DEBUG:teuthology.orchestra.run.smithi090:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-11-25T05:03:41.887 INFO:tasks.ceph.osd.7:Started 2023-11-25T05:03:41.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-25T05:03:42.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.019+0000 7fbf88844640 10 monclient: get_monmap_and_config 2023-11-25T05:03:42.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.019+0000 7fbf88844640 10 monclient: build_initial_monmap 2023-11-25T05:03:42.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.019+0000 7fbf88844640 10 monclient: monmap: 2023-11-25T05:03:42.021 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:03:42.021 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:42.021 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:03:42.020140+0000 2023-11-25T05:03:42.021 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:03:42.020140+0000 2023-11-25T05:03:42.021 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:42.021 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:42.021 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:03:42.021 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:03:42.021 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:03:42.021 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:03:42.021 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:03:42.021 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:03:42.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.019+0000 7fbf88844640 1 Processor -- start 2023-11-25T05:03:42.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.019+0000 7fbf88844640 1 -- start start 2023-11-25T05:03:42.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.019+0000 7fbf88844640 10 monclient: init 2023-11-25T05:03:42.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.020+0000 7fbf88844640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:42.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.020+0000 7fbf88844640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-11-25T05:03:42.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.020+0000 7fbf88844640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf840a8910 addr v1:172.21.15.7:6789/0 2023-11-25T05:03:42.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.020+0000 7fbf88844640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf840a9b70 addr v1:172.21.15.90:6789/0 2023-11-25T05:03:42.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.020+0000 7fbf88844640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbf8413f370 0x7fbf841426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:42.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.020+0000 7fbf88844640 10 monclient(hunting): picked mon.noname-c con 0x7fbf8413f370 addr v2:172.21.15.90:3300/0 2023-11-25T05:03:42.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.020+0000 7fbf88844640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf840a8cf0 con 0x7fbf840a8910 2023-11-25T05:03:42.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.020+0000 7fbf88844640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf840aabd0 con 0x7fbf840a9b70 2023-11-25T05:03:42.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.020+0000 7fbf88844640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:42.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.020+0000 7fbf88844640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf840aaeb0 con 0x7fbf8413f370 2023-11-25T05:03:42.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.020+0000 7fbf88844640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:42.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.020+0000 7fbf88844640 10 monclient(hunting): authenticate will time out at 1348.263792s 2023-11-25T05:03:42.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.021+0000 7fbf8259c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbf840a8910 0x7fbf8413ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53618/0 (socket says 172.21.15.7:53618) 2023-11-25T05:03:42.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.021+0000 7fbf8259c640 1 -- 172.21.15.7:0/3881438007 learned_addr learned my addr 172.21.15.7:0/3881438007 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:03:42.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.024+0000 7fbf8159a640 1 -- 172.21.15.7:0/3881438007 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619058817 0 0) 0x7fbf840a8cf0 con 0x7fbf840a8910 2023-11-25T05:03:42.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.024+0000 7fbf8159a640 10 monclient(hunting): _negotiate 2023-11-25T05:03:42.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.024+0000 7fbf8159a640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:42.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.024+0000 7fbf8159a640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:42.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.024+0000 7fbf8159a640 10 monclient(hunting): my global_id is 4152 2023-11-25T05:03:42.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.024+0000 7fbf8159a640 1 -- 172.21.15.7:0/3881438007 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf6c003520 con 0x7fbf840a8910 2023-11-25T05:03:42.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 1 -- 172.21.15.7:0/3881438007 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4225190801 0 0) 0x7fbf6c003520 con 0x7fbf840a8910 2023-11-25T05:03:42.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 1 -- 172.21.15.7:0/3881438007 >> v2:172.21.15.90:3300/0 conn(0x7fbf8413f370 msgr2=0x7fbf841426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:42.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 1 --2- 172.21.15.7:0/3881438007 >> v2:172.21.15.90:3300/0 conn(0x7fbf8413f370 0x7fbf841426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:42.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 1 -- 172.21.15.7:0/3881438007 >> v1:172.21.15.90:6789/0 conn(0x7fbf840a9b70 legacy=0x7fbf84141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:42.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient: _finish_hunting 0 2023-11-25T05:03:42.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 1 monclient: found mon.noname-a-legacy 2023-11-25T05:03:42.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:42.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.7:6789/0 2023-11-25T05:03:42.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 1 -- 172.21.15.7:0/3881438007 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf840a9720 con 0x7fbf840a8910 2023-11-25T05:03:42.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient: _finish_auth 0 2023-11-25T05:03:42.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient: _check_auth_tickets 2023-11-25T05:03:42.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:42.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 1 -- 172.21.15.7:0/3881438007 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4243585761 0 0) 0x7fbf74001670 con 0x7fbf840a8910 2023-11-25T05:03:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-11-25T05:03:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient: dump: 2023-11-25T05:03:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:03:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:42.031 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:03:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] but i'm connected to v1:172.21.15.7:6789/0, reconnecting 2023-11-25T05:03:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 1 -- 172.21.15.7:0/3881438007 >> v1:172.21.15.7:6789/0 conn(0x7fbf840a8910 legacy=0x7fbf8413ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-25T05:03:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 1 --2- 172.21.15.7:0/3881438007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c0039c0 0x7fbf6c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient(hunting): picked mon.a con 0x7fbf6c0039c0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 1 --2- 172.21.15.7:0/3881438007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf6c0047c0 0x7fbf6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient(hunting): picked mon.b con 0x7fbf6c0047c0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 1 --2- 172.21.15.7:0/3881438007 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbf6c0052a0 0x7fbf6c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient(hunting): picked mon.c con 0x7fbf6c0052a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 1 -- 172.21.15.7:0/3881438007 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf840aaeb0 con 0x7fbf6c0039c0 2023-11-25T05:03:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 1 -- 172.21.15.7:0/3881438007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c00c1f0 con 0x7fbf6c0047c0 2023-11-25T05:03:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:42.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 1 -- 172.21.15.7:0/3881438007 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c00c3b0 con 0x7fbf6c0052a0 2023-11-25T05:03:42.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:42.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.028+0000 7fbf8159a640 10 monclient(hunting): _finish_auth 0 2023-11-25T05:03:42.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.029+0000 7fbf82d9d640 1 --2- 172.21.15.7:0/3881438007 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbf6c0052a0 0x7fbf6c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:42.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.030+0000 7fbf82d9d640 10 monclient(hunting): get_auth_request con 0x7fbf6c0052a0 auth_method 0 2023-11-25T05:03:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.030+0000 7fbf82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.030+0000 7fbf82d9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.030+0000 7fbf82d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.030+0000 7fbf82d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.030+0000 7fbf82d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.030+0000 7fbf82d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-11-25T05:03:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.032+0000 7fbf8259c640 1 --2- 172.21.15.7:0/3881438007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c0039c0 0x7fbf6c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.032+0000 7fbf8259c640 10 monclient(hunting): get_auth_request con 0x7fbf6c0039c0 auth_method 0 2023-11-25T05:03:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.032+0000 7fbf8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.032+0000 7fbf8259c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.032+0000 7fbf8259c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:03:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf8259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-11-25T05:03:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf82d9d640 10 monclient(hunting): handle_auth_done global_id 4152 payload 293 2023-11-25T05:03:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf82d9d640 1 -- 172.21.15.7:0/3881438007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf6c0047c0 msgr2=0x7fbf6c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf82d9d640 1 --2- 172.21.15.7:0/3881438007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf6c0047c0 0x7fbf6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf82d9d640 1 -- 172.21.15.7:0/3881438007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c0039c0 msgr2=0x7fbf6c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf82d9d640 1 --2- 172.21.15.7:0/3881438007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c0039c0 0x7fbf6c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf82d9d640 10 monclient: _finish_hunting 0 2023-11-25T05:03:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf82d9d640 1 monclient: found mon.c 2023-11-25T05:03:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf82d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf82d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:03:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf82d9d640 1 -- 172.21.15.7:0/3881438007 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c003520 con 0x7fbf6c0052a0 2023-11-25T05:03:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf82d9d640 10 monclient: _finish_auth 0 2023-11-25T05:03:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf82d9d640 10 monclient: _check_auth_tickets 2023-11-25T05:03:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf82d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf88844640 5 monclient: authenticate success, global_id 4152 2023-11-25T05:03:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.033+0000 7fbf88844640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:03:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.034+0000 7fbf82d9d640 1 -- 172.21.15.7:0/3881438007 reap_dead start 2023-11-25T05:03:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.036+0000 7fbf82d9d640 1 --2- 172.21.15.7:0/3881438007 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbf6c0052a0 0x7fbf6c00b980 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c009e30 tx=0x7fbf6c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e8d04e85e47bbe1 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:42.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.041+0000 7fbf8159a640 1 -- 172.21.15.7:0/3881438007 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c02cd70 con 0x7fbf6c0052a0 2023-11-25T05:03:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.041+0000 7fbf8159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.041+0000 7fbf8159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:03:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.041+0000 7fbf8159a640 10 monclient: dump: 2023-11-25T05:03:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:03:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:42.046 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:03:42.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.041+0000 7fbf8159a640 1 -- 172.21.15.7:0/3881438007 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7c003990 con 0x7fbf6c0052a0 2023-11-25T05:03:42.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.041+0000 7fbf88844640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:03:42.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.041+0000 7fbf8159a640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:42.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.041+0000 7fbf88844640 10 monclient: get_monmap_and_config success 2023-11-25T05:03:42.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.041+0000 7fbf8159a640 1 -- 172.21.15.7:0/3881438007 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c003d20 con 0x7fbf6c0052a0 2023-11-25T05:03:42.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.041+0000 7fbf8159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:42.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.041+0000 7fbf88844640 4 set_mon_vals no callback set 2023-11-25T05:03:42.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.041+0000 7fbf8159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:03:42.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.041+0000 7fbf8159a640 10 monclient: dump: 2023-11-25T05:03:42.047 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:03:42.047 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:42.047 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:42.047 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:42.047 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:42.047 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:42.047 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:42.047 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:42.047 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:42.047 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:03:42.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.042+0000 7fbf88844640 10 monclient: shutdown 2023-11-25T05:03:42.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.042+0000 7fbf88844640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:42.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.042+0000 7fbf88844640 1 -- 172.21.15.7:0/3881438007 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbf6c0052a0 msgr2=0x7fbf6c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:42.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.042+0000 7fbf88844640 1 --2- 172.21.15.7:0/3881438007 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbf6c0052a0 0x7fbf6c00b980 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c009e30 tx=0x7fbf6c0057f0 comp rx=0 tx=0).stop 2023-11-25T05:03:42.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.042+0000 7fbf88844640 1 -- 172.21.15.7:0/3881438007 shutdown_connections 2023-11-25T05:03:42.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.042+0000 7fbf88844640 1 --2- 172.21.15.7:0/3881438007 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbf6c0052a0 0x7fbf6c00b980 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:42.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.042+0000 7fbf88844640 1 -- 172.21.15.7:0/3881438007 >> 172.21.15.7:0/3881438007 conn(0x7fbf840b7780 msgr2=0x7fbf840b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:42.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.042+0000 7fbf88844640 1 -- 172.21.15.7:0/3881438007 shutdown_connections 2023-11-25T05:03:42.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.042+0000 7fbf88844640 1 -- 172.21.15.7:0/3881438007 wait complete. 2023-11-25T05:03:42.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.043+0000 7fbf88844640 10 monclient: build_initial_monmap 2023-11-25T05:03:42.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.043+0000 7fbf88844640 10 monclient: monmap: 2023-11-25T05:03:42.048 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:03:42.048 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:03:42.048 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:03:42.048 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:03:42.048 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:42.048 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:42.049 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:03:42.049 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:03:42.049 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:03:42.049 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:03:42.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.044+0000 7fbf88844640 1 librados: starting msgr at 2023-11-25T05:03:42.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.044+0000 7fbf88844640 1 librados: starting objecter 2023-11-25T05:03:42.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.044+0000 7fbf88844640 1 Processor -- start 2023-11-25T05:03:42.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.044+0000 7fbf88844640 1 -- start start 2023-11-25T05:03:42.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.044+0000 7fbf88844640 1 librados: setting wanted keys 2023-11-25T05:03:42.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.044+0000 7fbf88844640 1 librados: calling monclient init 2023-11-25T05:03:42.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.044+0000 7fbf88844640 10 monclient: init 2023-11-25T05:03:42.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.044+0000 7fbf88844640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:42.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.044+0000 7fbf88844640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:03:42.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.044+0000 7fbf88844640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c0052a0 0x7fbf840b6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:42.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.044+0000 7fbf88844640 10 monclient(hunting): picked mon.noname-a con 0x7fbf6c0052a0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:42.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.044+0000 7fbf88844640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbf8406d940 0x7fbf84070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:42.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.045+0000 7fbf88844640 10 monclient(hunting): picked mon.noname-c con 0x7fbf8406d940 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:42.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.045+0000 7fbf88844640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf8406dd60 0x7fbf84071420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:42.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.045+0000 7fbf88844640 10 monclient(hunting): picked mon.noname-b con 0x7fbf8406dd60 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:42.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.045+0000 7fbf88844640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:42.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.045+0000 7fbf88844640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf841cc7e0 con 0x7fbf6c0052a0 2023-11-25T05:03:42.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.045+0000 7fbf88844640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:42.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.045+0000 7fbf88844640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8406e770 con 0x7fbf8406dd60 2023-11-25T05:03:42.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.045+0000 7fbf88844640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:42.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.045+0000 7fbf88844640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf841cb5c0 con 0x7fbf8406d940 2023-11-25T05:03:42.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.045+0000 7fbf88844640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:42.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.045+0000 7fbf88844640 10 monclient(hunting): authenticate will time out at 1348.287951s 2023-11-25T05:03:42.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.045+0000 7fbf81d9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbf8406d940 0x7fbf84070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:42.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.045+0000 7fbf8259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c0052a0 0x7fbf840b6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:42.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf8259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c0052a0 0x7fbf840b6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56356/0 (socket says 172.21.15.7:56356) 2023-11-25T05:03:42.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf8259c640 1 -- 172.21.15.7:0/3323497440 learned_addr learned my addr 172.21.15.7:0/3323497440 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:03:42.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf8259c640 10 monclient(hunting): get_auth_request con 0x7fbf6c0052a0 auth_method 0 2023-11-25T05:03:42.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:42.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf8259c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:42.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf8259c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:42.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf81d9b640 10 monclient(hunting): get_auth_request con 0x7fbf8406d940 auth_method 0 2023-11-25T05:03:42.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf81d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:42.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf81d9b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:42.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf81d9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:42.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:42.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:42.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:42.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf81d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:42.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf81d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:42.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.046+0000 7fbf81d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:42.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.051+0000 7fbf8259c640 10 monclient(hunting): handle_auth_done global_id 4155 payload 824 2023-11-25T05:03:42.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.051+0000 7fbf8259c640 1 -- 172.21.15.7:0/3323497440 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbf8406d940 msgr2=0x7fbf84070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:42.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.051+0000 7fbf8259c640 1 --2- 172.21.15.7:0/3323497440 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbf8406d940 0x7fbf84070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:42.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.051+0000 7fbf8259c640 1 -- 172.21.15.7:0/3323497440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf8406dd60 msgr2=0x7fbf84071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:42.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.051+0000 7fbf8259c640 1 --2- 172.21.15.7:0/3323497440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf8406dd60 0x7fbf84071420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:42.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.051+0000 7fbf8259c640 10 monclient: _finish_hunting 0 2023-11-25T05:03:42.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.052+0000 7fbf8259c640 1 monclient: found mon.noname-a 2023-11-25T05:03:42.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.052+0000 7fbf8259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:42.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.052+0000 7fbf8259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:03:42.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.052+0000 7fbf8259c640 1 -- 172.21.15.7:0/3323497440 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8406e250 con 0x7fbf6c0052a0 2023-11-25T05:03:42.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.052+0000 7fbf81d9b640 1 --2- 172.21.15.7:0/3323497440 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbf8406d940 0x7fbf84070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-25T05:03:42.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.054+0000 7fbf8259c640 1 --2- 172.21.15.7:0/3323497440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c0052a0 0x7fbf840b6e80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbf74007290 tx=0x7fbf74007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:42.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.054+0000 7fbf72ffd640 10 client.?.objecter ms_handle_connect 0x7fbf6c0052a0 2023-11-25T05:03:42.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.054+0000 7fbf72ffd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:03:42.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.058+0000 7fbf72ffd640 1 -- 172.21.15.7:0/3323497440 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7400f7b0 con 0x7fbf6c0052a0 2023-11-25T05:03:42.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.058+0000 7fbf72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:42.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.058+0000 7fbf72ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:03:42.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.058+0000 7fbf72ffd640 10 monclient: dump: 2023-11-25T05:03:42.062 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:03:42.062 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:42.062 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:42.063 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:42.063 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:42.063 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:42.063 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:42.063 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:42.063 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:42.063 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:03:42.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.058+0000 7fbf72ffd640 10 monclient: _finish_auth 0 2023-11-25T05:03:42.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.058+0000 7fbf72ffd640 10 monclient: _check_auth_tickets 2023-11-25T05:03:42.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.058+0000 7fbf72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:42.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.058+0000 7fbf72ffd640 1 -- 172.21.15.7:0/3323497440 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7400f930 con 0x7fbf6c0052a0 2023-11-25T05:03:42.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.058+0000 7fbf72ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:42.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.058+0000 7fbf72ffd640 1 -- 172.21.15.7:0/3323497440 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7400fce0 con 0x7fbf6c0052a0 2023-11-25T05:03:42.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.059+0000 7fbf72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:42.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.059+0000 7fbf72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:03:42.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.059+0000 7fbf72ffd640 10 monclient: dump: 2023-11-25T05:03:42.064 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:03:42.064 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:42.064 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:42.064 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:42.064 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:42.064 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:42.064 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:42.064 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:42.064 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:42.064 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:03:42.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.059+0000 7fbf88844640 5 monclient: authenticate success, global_id 4155 2023-11-25T05:03:42.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.059+0000 7fbf88844640 10 monclient: _renew_subs 2023-11-25T05:03:42.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.059+0000 7fbf88844640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:03:42.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.059+0000 7fbf88844640 1 -- 172.21.15.7:0/3323497440 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf84071960 con 0x7fbf6c0052a0 2023-11-25T05:03:42.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.059+0000 7fbf80d99640 4 set_mon_vals no callback set 2023-11-25T05:03:42.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.059+0000 7fbf88844640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:03:42.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.059+0000 7fbf88844640 10 monclient: _renew_subs 2023-11-25T05:03:42.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.059+0000 7fbf88844640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:03:42.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.059+0000 7fbf88844640 1 -- 172.21.15.7:0/3323497440 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf841de680 con 0x7fbf6c0052a0 2023-11-25T05:03:42.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.059+0000 7fbf88844640 1 librados: init done 2023-11-25T05:03:42.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.060+0000 7fbf88844640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:42.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.060+0000 7fbf88844640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:42.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.060+0000 7fbf88844640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:03:42.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.060+0000 7fbf88844640 1 -- 172.21.15.7:0/3323497440 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf8406e440 con 0x7fbf6c0052a0 2023-11-25T05:03:42.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.265+0000 7fbf72ffd640 1 -- 172.21.15.7:0/3323497440 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 2) v1 ==== 44567+0+0 (secure 0 0 0) 0x7fbf7401d460 con 0x7fbf6c0052a0 2023-11-25T05:03:42.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.306+0000 7fbf72ffd640 1 -- 172.21.15.7:0/3323497440 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fbf7401d710 con 0x7fbf6c0052a0 2023-11-25T05:03:42.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.306+0000 7fbf72ffd640 10 client.4155.objecter ms_dispatch 0x7fbf84000b90 osd_map(10..10 src has 1..10) v4 2023-11-25T05:03:42.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.306+0000 7fbf72ffd640 3 client.4155.objecter handle_osd_map got epochs [10,10] > 0 2023-11-25T05:03:42.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.306+0000 7fbf72ffd640 3 client.4155.objecter handle_osd_map decoding full epoch 10 2023-11-25T05:03:42.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.306+0000 7fbf72ffd640 20 client.4155.objecter dump_active .. 0 homeless 2023-11-25T05:03:42.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.312+0000 7fbf72ffd640 1 -- 172.21.15.7:0/3323497440 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fbf7401b950 con 0x7fbf6c0052a0 2023-11-25T05:03:42.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.312+0000 7fbf72ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:42.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.312+0000 7fbf72ffd640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:03:42.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.438+0000 7fbf88844640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:03:42.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.438+0000 7fbf88844640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:03:42.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.438+0000 7fbf88844640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:03:42.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.438+0000 7fbf88844640 1 -- 172.21.15.7:0/3323497440 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf84075b60 con 0x7fbf6c0052a0 2023-11-25T05:03:42.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.698+0000 7fbf72ffd640 1 -- 172.21.15.7:0/3323497440 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fbf740263e0 con 0x7fbf6c0052a0 2023-11-25T05:03:42.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.698+0000 7fbf72ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:03:42.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.698+0000 7fbf72ffd640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:03:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:03:42.700 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":10,"fsid":"59cf1eff-816d-4eda-b513-a2de7ea218f7","created":"2023-11-25T05:03:35.598582+0000","modified":"2023-11-25T05:03:41.565581+0000","last_up_change":"0.000000","last_in_change":"2023-11-25T05:03:41.565581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c87807e-7b82-4866-ba36-60b44a25f317","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0e252ab8-9524-4ccc-b08f-d95bbf05fbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccda96b8-aa9e-4c99-8d84-bba19ca0483b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e4b8700a-2e28-4e69-8577-465666b1b051","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"36c88a76-0467-4e37-85b4-dfd5953855a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ebf452a0-1b17-488a-bbea-2e953c932ac7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5aaf35c3-5c06-45e5-b788-8d3207c47cd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d50dd23a-f864-4593-ac00-4d8afed21dd0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-25T05:03:42.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.699+0000 7fbf88844640 10 librados: watch_flush enter 2023-11-25T05:03:42.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.700+0000 7fbf88844640 10 librados: watch_flush exit 2023-11-25T05:03:42.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.700+0000 7fbf88844640 20 client.4155.objecter shutdown clearing up homeless session... 2023-11-25T05:03:42.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.700+0000 7fbf88844640 10 client.4155.objecter successfully canceled tick 2023-11-25T05:03:42.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.700+0000 7fbf88844640 10 monclient: shutdown 2023-11-25T05:03:42.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.700+0000 7fbf88844640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:42.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.700+0000 7fbf88844640 1 -- 172.21.15.7:0/3323497440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c0052a0 msgr2=0x7fbf840b6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:42.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.700+0000 7fbf88844640 1 --2- 172.21.15.7:0/3323497440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c0052a0 0x7fbf840b6e80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbf74007290 tx=0x7fbf74007750 comp rx=0 tx=0).stop 2023-11-25T05:03:42.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.700+0000 7fbf88844640 1 -- 172.21.15.7:0/3323497440 shutdown_connections 2023-11-25T05:03:42.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.700+0000 7fbf88844640 1 --2- 172.21.15.7:0/3323497440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf8406dd60 0x7fbf84071420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:42.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.700+0000 7fbf88844640 1 --2- 172.21.15.7:0/3323497440 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbf8406d940 0x7fbf84070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:42.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.700+0000 7fbf88844640 1 --2- 172.21.15.7:0/3323497440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c0052a0 0x7fbf840b6e80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:42.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.700+0000 7fbf88844640 1 -- 172.21.15.7:0/3323497440 >> 172.21.15.7:0/3323497440 conn(0x7fbf840b7780 msgr2=0x7fbf840b7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:42.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.700+0000 7fbf88844640 1 -- 172.21.15.7:0/3323497440 shutdown_connections 2023-11-25T05:03:42.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.700+0000 7fbf88844640 1 -- 172.21.15.7:0/3323497440 wait complete. 2023-11-25T05:03:42.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:03:42.700+0000 7fbf88844640 1 librados: shutdown 2023-11-25T05:03:42.717 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-25T05:03:42.718 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-25T05:03:43.231 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:43.230+0000 7fdc6e695640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-25T05:03:43.523 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-25T05:03:43.571 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:03:43.570+0000 7fdc6e695640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-25T05:03:43.678 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.676+0000 7f9da652c640 10 monclient: get_monmap_and_config 2023-11-25T05:03:43.678 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.676+0000 7f9da652c640 10 monclient: build_initial_monmap 2023-11-25T05:03:43.678 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.676+0000 7f9da652c640 10 monclient: monmap: 2023-11-25T05:03:43.678 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-11-25T05:03:43.678 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:43.678 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:03:43.677464+0000 2023-11-25T05:03:43.678 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:03:43.677464+0000 2023-11-25T05:03:43.678 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:43.678 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:43.678 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:03:43.678 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:03:43.679 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:03:43.679 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:03:43.679 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:03:43.679 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:43.681 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.680+0000 7f9da652c640 1 Processor -- start 2023-11-25T05:03:43.681 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.680+0000 7f9da652c640 1 -- start start 2023-11-25T05:03:43.682 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.680+0000 7f9da652c640 10 monclient: init 2023-11-25T05:03:43.683 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.681+0000 7f9da652c640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:43.683 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.681+0000 7f9da652c640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-11-25T05:03:43.684 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.681+0000 7f9da652c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9da00a8a00 addr v1:172.21.15.7:6789/0 2023-11-25T05:03:43.684 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.682+0000 7f9da652c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9da0146150 addr v1:172.21.15.90:6789/0 2023-11-25T05:03:43.684 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.682+0000 7f9da652c640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9da00a8de0 0x7f9da013f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:43.684 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.682+0000 7f9da652c640 10 monclient(hunting): picked mon.noname-c con 0x7f9da00a8de0 addr v2:172.21.15.90:3300/0 2023-11-25T05:03:43.684 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.682+0000 7f9da652c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da01455b0 con 0x7f9da00a8a00 2023-11-25T05:03:43.684 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.682+0000 7f9da652c640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da00aa970 con 0x7f9da0146150 2023-11-25T05:03:43.684 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.682+0000 7f9da652c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:43.684 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.682+0000 7f9da652c640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da00aac50 con 0x7f9da00a8de0 2023-11-25T05:03:43.684 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.682+0000 7f9da652c640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:43.684 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.682+0000 7f9da652c640 10 monclient(hunting): authenticate will time out at 1349.925314s 2023-11-25T05:03:43.685 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.683+0000 7f9da4ac9640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9da00a8de0 0x7f9da013f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:43.685 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.683+0000 7f9da4ac9640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9da00a8de0 0x7f9da013f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:40760/0 (socket says 172.21.15.7:40760) 2023-11-25T05:03:43.685 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.683+0000 7f9da4ac9640 1 -- 172.21.15.7:0/2664244074 learned_addr learned my addr 172.21.15.7:0/2664244074 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:03:43.685 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.683+0000 7f9da4ac9640 10 monclient(hunting): get_auth_request con 0x7f9da00a8de0 auth_method 0 2023-11-25T05:03:43.685 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.683+0000 7f9da4ac9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:43.685 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.683+0000 7f9da4ac9640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:43.685 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.683+0000 7f9da4ac9640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:43.686 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.684+0000 7f9da4ac9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:43.686 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.684+0000 7f9da4ac9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:43.686 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.684+0000 7f9da4ac9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:43.689 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.687+0000 7f9da4ac9640 10 monclient(hunting): handle_auth_done global_id 4168 payload 274 2023-11-25T05:03:43.689 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.687+0000 7f9da4ac9640 1 -- 172.21.15.7:0/2664244074 >> v1:172.21.15.90:6789/0 conn(0x7f9da0146150 legacy=0x7f9da013eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:43.689 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.687+0000 7f9da4ac9640 1 -- 172.21.15.7:0/2664244074 >> v1:172.21.15.7:6789/0 conn(0x7f9da00a8a00 legacy=0x7f9da0145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:03:43.689 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.687+0000 7f9da4ac9640 10 monclient: _finish_hunting 0 2023-11-25T05:03:43.689 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.687+0000 7f9da4ac9640 1 monclient: found mon.noname-c 2023-11-25T05:03:43.689 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.687+0000 7f9da4ac9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:43.689 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.687+0000 7f9da4ac9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:03:43.689 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.687+0000 7f9da4ac9640 1 -- 172.21.15.7:0/2664244074 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da00aaf30 con 0x7f9da00a8de0 2023-11-25T05:03:43.694 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.693+0000 7f9da4ac9640 1 --2- 172.21.15.7:0/2664244074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9da00a8de0 0x7f9da013f3b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9d98009e50 tx=0x7f9d9802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fe104087c1c385e server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:43.705 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9d9effd640 1 -- 172.21.15.7:0/2664244074 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9802ca80 con 0x7f9da00a8de0 2023-11-25T05:03:43.706 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9d9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:43.706 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9d9effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:03:43.706 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9d9effd640 10 monclient: dump: 2023-11-25T05:03:43.706 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:03:43.706 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:43.706 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:43.706 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:43.706 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:43.706 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:43.706 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:43.707 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:43.707 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:43.707 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:43.707 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9d9effd640 10 monclient: _finish_auth 0 2023-11-25T05:03:43.707 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9d9effd640 10 monclient: _check_auth_tickets 2023-11-25T05:03:43.707 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9d9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:43.707 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9d9effd640 1 -- 172.21.15.7:0/2664244074 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9802cbe0 con 0x7f9da00a8de0 2023-11-25T05:03:43.707 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9d9effd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:43.707 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9d9effd640 1 -- 172.21.15.7:0/2664244074 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d980343f0 con 0x7f9da00a8de0 2023-11-25T05:03:43.707 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9d9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:43.708 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9d9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:03:43.708 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9d9effd640 10 monclient: dump: 2023-11-25T05:03:43.708 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:03:43.708 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:43.708 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:43.708 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:43.708 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:43.708 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:43.708 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:43.708 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:43.708 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:43.708 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:43.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9da652c640 5 monclient: authenticate success, global_id 4168 2023-11-25T05:03:43.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9da652c640 10 monclient: get_monmap_and_config success 2023-11-25T05:03:43.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.701+0000 7f9da652c640 4 set_mon_vals no callback set 2023-11-25T05:03:43.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.702+0000 7f9da652c640 10 monclient: shutdown 2023-11-25T05:03:43.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.702+0000 7f9da652c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:43.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.702+0000 7f9da652c640 1 -- 172.21.15.7:0/2664244074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9da00a8de0 msgr2=0x7f9da013f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:43.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.702+0000 7f9da652c640 1 --2- 172.21.15.7:0/2664244074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9da00a8de0 0x7f9da013f3b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9d98009e50 tx=0x7f9d9802bee0 comp rx=0 tx=0).stop 2023-11-25T05:03:43.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.702+0000 7f9da652c640 1 -- 172.21.15.7:0/2664244074 shutdown_connections 2023-11-25T05:03:43.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.702+0000 7f9da652c640 1 --2- 172.21.15.7:0/2664244074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9da00a8de0 0x7f9da013f3b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:43.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.702+0000 7f9da652c640 1 -- 172.21.15.7:0/2664244074 >> 172.21.15.7:0/2664244074 conn(0x7f9da0066e80 msgr2=0x7f9da00b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:43.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.702+0000 7f9da652c640 1 -- 172.21.15.7:0/2664244074 shutdown_connections 2023-11-25T05:03:43.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.702+0000 7f9da652c640 1 -- 172.21.15.7:0/2664244074 wait complete. 2023-11-25T05:03:43.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.703+0000 7f9da652c640 10 monclient: build_initial_monmap 2023-11-25T05:03:43.710 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.703+0000 7f9da652c640 10 monclient: monmap: 2023-11-25T05:03:43.710 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-11-25T05:03:43.710 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:03:43.710 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2023-11-25T05:03:43.710 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2023-11-25T05:03:43.710 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:43.710 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:43.710 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:03:43.710 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:03:43.710 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:03:43.710 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:43.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.704+0000 7f9da652c640 1 librados: starting msgr at 2023-11-25T05:03:43.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.704+0000 7f9da652c640 1 librados: starting objecter 2023-11-25T05:03:43.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.704+0000 7f9da652c640 1 Processor -- start 2023-11-25T05:03:43.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.704+0000 7f9da652c640 1 -- start start 2023-11-25T05:03:43.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.704+0000 7f9da652c640 1 librados: setting wanted keys 2023-11-25T05:03:43.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.704+0000 7f9da652c640 1 librados: calling monclient init 2023-11-25T05:03:43.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.704+0000 7f9da652c640 10 monclient: init 2023-11-25T05:03:43.714 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.705+0000 7f9da652c640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:43.714 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.705+0000 7f9da652c640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-25T05:03:43.714 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.706+0000 7f9da652c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9da00a8a00 0x7f9da00b6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:43.715 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.706+0000 7f9da652c640 10 monclient(hunting): picked mon.noname-b con 0x7f9da00a8a00 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:43.715 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.706+0000 7f9da652c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9da00a8de0 0x7f9da013a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:43.715 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.706+0000 7f9da652c640 10 monclient(hunting): picked mon.noname-c con 0x7f9da00a8de0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:43.715 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.706+0000 7f9da652c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da0146150 0x7f9da013b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:43.715 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.706+0000 7f9da652c640 10 monclient(hunting): picked mon.noname-a con 0x7f9da0146150 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:43.715 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.706+0000 7f9da652c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:43.715 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.706+0000 7f9da652c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da013abb0 con 0x7f9da0146150 2023-11-25T05:03:43.715 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.706+0000 7f9da652c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:43.715 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.707+0000 7f9da652c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da013c3a0 con 0x7f9da00a8a00 2023-11-25T05:03:43.715 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.707+0000 7f9da652c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:43.715 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.707+0000 7f9da652c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da013ffc0 con 0x7f9da00a8de0 2023-11-25T05:03:43.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.707+0000 7f9da652c640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:43.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.707+0000 7f9da652c640 10 monclient(hunting): authenticate will time out at 1349.950133s 2023-11-25T05:03:43.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.708+0000 7f9da4ac9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da0146150 0x7f9da013b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:43.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.709+0000 7f9d9ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9da00a8a00 0x7f9da00b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:43.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.709+0000 7f9da4ac9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da0146150 0x7f9da013b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56362/0 (socket says 172.21.15.7:56362) 2023-11-25T05:03:43.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.709+0000 7f9d9ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9da00a8a00 0x7f9da00b6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:40772/0 (socket says 172.21.15.7:40772) 2023-11-25T05:03:43.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.709+0000 7f9d9ffff640 1 -- 172.21.15.7:0/3022252568 learned_addr learned my addr 172.21.15.7:0/3022252568 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:03:43.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.709+0000 7f9d9ffff640 10 monclient(hunting): get_auth_request con 0x7f9da00a8a00 auth_method 0 2023-11-25T05:03:43.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.709+0000 7f9d9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:43.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.709+0000 7f9d9ffff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:43.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.709+0000 7f9d9ffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:43.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.709+0000 7f9da4ac9640 10 monclient(hunting): get_auth_request con 0x7f9da0146150 auth_method 0 2023-11-25T05:03:43.717 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.709+0000 7f9da4ac9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:43.717 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.709+0000 7f9da4ac9640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:43.717 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.709+0000 7f9da4ac9640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:43.717 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.710+0000 7f9d9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:43.717 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.710+0000 7f9d9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:43.717 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.710+0000 7f9d9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:43.717 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.710+0000 7f9da4ac9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:43.717 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.710+0000 7f9da4ac9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:43.717 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.710+0000 7f9da4ac9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:43.718 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.714+0000 7f9d9ffff640 10 monclient(hunting): handle_auth_done global_id 4171 payload 824 2023-11-25T05:03:43.718 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.714+0000 7f9d9ffff640 1 -- 172.21.15.7:0/3022252568 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9da00a8de0 msgr2=0x7f9da013a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:43.718 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.714+0000 7f9d9ffff640 1 --2- 172.21.15.7:0/3022252568 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9da00a8de0 0x7f9da013a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:43.719 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.714+0000 7f9d9ffff640 1 -- 172.21.15.7:0/3022252568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da0146150 msgr2=0x7f9da013b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:43.719 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.714+0000 7f9d9ffff640 1 --2- 172.21.15.7:0/3022252568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da0146150 0x7f9da013b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:43.719 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.714+0000 7f9d9ffff640 10 monclient: _finish_hunting 0 2023-11-25T05:03:43.719 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.714+0000 7f9d9ffff640 1 monclient: found mon.noname-b 2023-11-25T05:03:43.719 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.714+0000 7f9d9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:43.719 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.714+0000 7f9d9ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:03:43.719 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.714+0000 7f9d9ffff640 1 -- 172.21.15.7:0/3022252568 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d98009b30 con 0x7f9da00a8a00 2023-11-25T05:03:43.720 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.718+0000 7f9d9ffff640 1 --2- 172.21.15.7:0/3022252568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9da00a8a00 0x7f9da00b6720 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9d9000c440 tx=0x7f9d9000c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:43.720 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.718+0000 7f9d9cff9640 10 client.?.objecter ms_handle_connect 0x7f9da00a8a00 2023-11-25T05:03:43.720 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.718+0000 7f9d9cff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:03:43.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.723+0000 7f9d9cff9640 1 -- 172.21.15.7:0/3022252568 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d90005480 con 0x7f9da00a8a00 2023-11-25T05:03:43.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.723+0000 7f9d9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:43.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.723+0000 7f9d9cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:03:43.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.723+0000 7f9d9cff9640 10 monclient: dump: 2023-11-25T05:03:43.729 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:03:43.729 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:43.729 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:43.729 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:43.729 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:43.730 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:43.730 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:43.730 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:43.730 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:43.730 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:43.730 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.723+0000 7f9d9cff9640 10 monclient: _finish_auth 0 2023-11-25T05:03:43.730 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.723+0000 7f9d9cff9640 10 monclient: _check_auth_tickets 2023-11-25T05:03:43.730 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.723+0000 7f9d9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:43.730 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.723+0000 7f9d9cff9640 1 -- 172.21.15.7:0/3022252568 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d900035f0 con 0x7f9da00a8a00 2023-11-25T05:03:43.730 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.723+0000 7f9d9cff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:43.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.723+0000 7f9d9e7fc640 4 set_mon_vals no callback set 2023-11-25T05:03:43.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.724+0000 7f9d9cff9640 1 -- 172.21.15.7:0/3022252568 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d900039c0 con 0x7f9da00a8a00 2023-11-25T05:03:43.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.724+0000 7f9da652c640 5 monclient: authenticate success, global_id 4171 2023-11-25T05:03:43.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.724+0000 7f9da652c640 10 monclient: _renew_subs 2023-11-25T05:03:43.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.724+0000 7f9da652c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:43.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.724+0000 7f9da652c640 1 -- 172.21.15.7:0/3022252568 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da00b70f0 con 0x7f9da00a8a00 2023-11-25T05:03:43.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.725+0000 7f9d9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:43.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.725+0000 7f9d9cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:03:43.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.725+0000 7f9d9cff9640 10 monclient: dump: 2023-11-25T05:03:43.731 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:03:43.731 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:43.731 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:43.731 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:43.731 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:43.731 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:43.732 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:43.732 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:43.732 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:43.732 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:43.732 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.725+0000 7f9da652c640 10 client.4171.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:03:43.732 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.725+0000 7f9da652c640 10 monclient: _renew_subs 2023-11-25T05:03:43.732 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.725+0000 7f9da652c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:43.732 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.725+0000 7f9da652c640 1 -- 172.21.15.7:0/3022252568 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da01408a0 con 0x7f9da00a8a00 2023-11-25T05:03:43.732 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.725+0000 7f9da652c640 1 librados: init done 2023-11-25T05:03:43.732 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.726+0000 7f9da652c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:43.733 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.726+0000 7f9da652c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:43.733 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.727+0000 7f9da652c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:43.733 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.727+0000 7f9da652c640 1 -- 172.21.15.7:0/3022252568 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da0069c00 con 0x7f9da00a8a00 2023-11-25T05:03:43.763 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.761+0000 7f9d9cff9640 1 -- 172.21.15.7:0/3022252568 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 3) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f9d90003b20 con 0x7f9da00a8a00 2023-11-25T05:03:43.763 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.762+0000 7f9d9cff9640 1 --2- 172.21.15.7:0/3022252568 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f9d8c06c620 0x7f9d8c06eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:43.764 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.762+0000 7f9d9f7fe640 1 --2- 172.21.15.7:0/3022252568 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f9d8c06c620 0x7f9d8c06eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:43.764 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.762+0000 7f9d9f7fe640 10 monclient: get_auth_request con 0x7f9d8c06c620 auth_method 0 2023-11-25T05:03:43.764 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.763+0000 7f9d9f7fe640 1 --2- 172.21.15.7:0/3022252568 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f9d8c06c620 0x7f9d8c06eae0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9d94006fd0 tx=0x7f9d94006bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:43.764 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.763+0000 7f9d9cff9640 10 client.4171.objecter ms_handle_connect 0x7f9d8c06c620 2023-11-25T05:03:43.769 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.766+0000 7f9d9cff9640 1 -- 172.21.15.7:0/3022252568 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f9d900010c0 con 0x7f9da00a8a00 2023-11-25T05:03:43.769 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.766+0000 7f9d9cff9640 10 client.4171.objecter ms_dispatch 0x7f9da0000b90 osd_map(10..10 src has 1..10) v4 2023-11-25T05:03:43.769 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.766+0000 7f9d9cff9640 3 client.4171.objecter handle_osd_map got epochs [10,10] > 0 2023-11-25T05:03:43.769 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.766+0000 7f9d9cff9640 3 client.4171.objecter handle_osd_map decoding full epoch 10 2023-11-25T05:03:43.769 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:43.766+0000 7f9d9cff9640 20 client.4171.objecter dump_active .. 0 homeless 2023-11-25T05:03:44.190 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.189+0000 7f9d9cff9640 1 -- 172.21.15.7:0/3022252568 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9d900573a0 con 0x7f9da00a8a00 2023-11-25T05:03:44.191 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.189+0000 7f9d9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:44.191 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.189+0000 7f9d9cff9640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:03:44.557 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.555+0000 7f9da652c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:03:44.557 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.556+0000 7f9da652c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:03:44.557 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.556+0000 7f9da652c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:44.557 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.556+0000 7f9da652c640 1 -- 172.21.15.7:0/3022252568 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9da0138b50 con 0x7f9da00a8a00 2023-11-25T05:03:44.785 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.783+0000 7f9d9cff9640 1 -- 172.21.15.7:0/3022252568 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f9d90017090 con 0x7f9da00a8a00 2023-11-25T05:03:44.785 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.783+0000 7f9d9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:03:44.785 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.783+0000 7f9d9cff9640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:03:44.786 INFO:teuthology.misc.health.smithi007.stdout: 2023-11-25T05:03:44.786 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"59cf1eff-816d-4eda-b513-a2de7ea218f7","created":"2023-11-25T05:03:35.598582+0000","modified":"2023-11-25T05:03:41.565581+0000","last_up_change":"0.000000","last_in_change":"2023-11-25T05:03:41.565581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c87807e-7b82-4866-ba36-60b44a25f317","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0e252ab8-9524-4ccc-b08f-d95bbf05fbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccda96b8-aa9e-4c99-8d84-bba19ca0483b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e4b8700a-2e28-4e69-8577-465666b1b051","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"36c88a76-0467-4e37-85b4-dfd5953855a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ebf452a0-1b17-488a-bbea-2e953c932ac7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5aaf35c3-5c06-45e5-b788-8d3207c47cd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d50dd23a-f864-4593-ac00-4d8afed21dd0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-25T05:03:44.788 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.786+0000 7f9da652c640 10 librados: watch_flush enter 2023-11-25T05:03:44.789 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.786+0000 7f9da652c640 10 librados: watch_flush exit 2023-11-25T05:03:44.789 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.786+0000 7f9da652c640 20 client.4171.objecter shutdown clearing up homeless session... 2023-11-25T05:03:44.789 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.786+0000 7f9da652c640 10 client.4171.objecter successfully canceled tick 2023-11-25T05:03:44.789 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.787+0000 7f9da652c640 1 -- 172.21.15.7:0/3022252568 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f9d8c06c620 msgr2=0x7f9d8c06eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:44.789 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.787+0000 7f9da652c640 1 --2- 172.21.15.7:0/3022252568 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f9d8c06c620 0x7f9d8c06eae0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9d94006fd0 tx=0x7f9d94006bc0 comp rx=0 tx=0).stop 2023-11-25T05:03:44.789 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.787+0000 7f9da652c640 10 monclient: shutdown 2023-11-25T05:03:44.789 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.787+0000 7f9da652c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:44.789 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.787+0000 7f9da652c640 1 -- 172.21.15.7:0/3022252568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9da00a8a00 msgr2=0x7f9da00b6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:44.789 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.787+0000 7f9da652c640 1 --2- 172.21.15.7:0/3022252568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9da00a8a00 0x7f9da00b6720 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9d9000c440 tx=0x7f9d9000c900 comp rx=0 tx=0).stop 2023-11-25T05:03:44.789 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.787+0000 7f9da652c640 1 -- 172.21.15.7:0/3022252568 shutdown_connections 2023-11-25T05:03:44.789 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.787+0000 7f9da652c640 1 --2- 172.21.15.7:0/3022252568 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f9d8c06c620 0x7f9d8c06eae0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:44.789 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.787+0000 7f9da652c640 1 --2- 172.21.15.7:0/3022252568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da0146150 0x7f9da013b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:44.790 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.787+0000 7f9da652c640 1 --2- 172.21.15.7:0/3022252568 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9da00a8de0 0x7f9da013a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:44.790 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.787+0000 7f9da652c640 1 --2- 172.21.15.7:0/3022252568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9da00a8a00 0x7f9da00b6720 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:44.790 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.787+0000 7f9da652c640 1 -- 172.21.15.7:0/3022252568 >> 172.21.15.7:0/3022252568 conn(0x7f9da0066e80 msgr2=0x7f9da0137c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:44.790 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.787+0000 7f9da652c640 1 -- 172.21.15.7:0/3022252568 shutdown_connections 2023-11-25T05:03:44.790 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.787+0000 7f9da652c640 1 -- 172.21.15.7:0/3022252568 wait complete. 2023-11-25T05:03:44.790 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:44.787+0000 7f9da652c640 1 librados: shutdown 2023-11-25T05:03:44.806 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-25T05:03:49.970 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:08.002 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:50.033 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:08.065 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:50.087 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:08.138 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:50.146 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:08.197 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:50.162 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:08.206 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:50.176 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:08.226 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:50.223 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:08.267 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:50.237 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:08.287 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:50.414 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:08.499 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:50.476 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:08.562 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:50.477 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:08.562 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:50.549 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:08.634 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:50.611 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-25T05:03:50.604+0000 5dc2340 -1 Falling back to public interface 2023-11-25T05:03:50.715 INFO:tasks.ceph.osd.5.smithi090.stderr:2023-11-25T05:03:50.708+0000 5dc2340 -1 Falling back to public interface 2023-11-25T05:03:50.797 INFO:tasks.ceph.osd.6.smithi090.stderr:2023-11-25T05:03:50.790+0000 5dc2340 -1 Falling back to public interface 2023-11-25T05:03:50.808 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:03:50.798+0000 5dc2340 -1 Falling back to public interface 2023-11-25T05:03:50.852 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:08.928 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:50.855 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:08.943 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:50.914 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:08.990 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:50.920 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:09.008 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.022 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-25T05:03:51.015+0000 5dc2340 -1 Falling back to public interface 2023-11-25T05:03:51.045 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:09.078 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.086 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:03:51.078+0000 5dc2340 -1 Falling back to public interface 2023-11-25T05:03:51.139 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:09.190 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.248 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:09.292 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.280 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:09.330 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.397 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:09.429 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.457 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:09.542 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.473 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-11-25T05:03:51.463+0000 5dc2340 -1 Falling back to public interface 2023-11-25T05:03:51.495 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:09.544 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.517 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-25T05:03:51.509+0000 5dc2340 -1 Falling back to public interface 2023-11-25T05:03:51.537 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:09.622 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.603 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:09.647 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.611 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-25T05:03:51.645 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:09.696 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.703 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.702+0000 7f64dc790640 10 monclient: get_monmap_and_config 2023-11-25T05:03:51.704 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.702+0000 7f64dc790640 10 monclient: build_initial_monmap 2023-11-25T05:03:51.704 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.702+0000 7f64dc790640 10 monclient: monmap: 2023-11-25T05:03:51.704 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-11-25T05:03:51.704 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:51.704 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:03:51.703342+0000 2023-11-25T05:03:51.704 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:03:51.703342+0000 2023-11-25T05:03:51.704 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:51.704 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:51.705 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:03:51.705 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:03:51.705 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:03:51.705 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:03:51.705 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:03:51.705 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:51.707 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.703+0000 7f64dc790640 1 Processor -- start 2023-11-25T05:03:51.707 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.703+0000 7f64dc790640 1 -- start start 2023-11-25T05:03:51.707 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.703+0000 7f64dc790640 10 monclient: init 2023-11-25T05:03:51.707 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.703+0000 7f64dc790640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:51.707 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.703+0000 7f64dc790640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-11-25T05:03:51.707 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.703+0000 7f64dc790640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f64d40a89e0 0x7f64d413ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:51.707 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.703+0000 7f64dc790640 10 monclient(hunting): picked mon.noname-b con 0x7f64d40a89e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:51.708 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.703+0000 7f64dc790640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64d413f460 addr v1:172.21.15.7:6789/0 2023-11-25T05:03:51.708 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.703+0000 7f64dc790640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64d41428e0 addr v1:172.21.15.90:6789/0 2023-11-25T05:03:51.708 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.703+0000 7f64dc790640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d40aa770 con 0x7f64d413f460 2023-11-25T05:03:51.708 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.703+0000 7f64dc790640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d413f860 con 0x7f64d41428e0 2023-11-25T05:03:51.708 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.703+0000 7f64dc790640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:51.708 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.703+0000 7f64dc790640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d4142d70 con 0x7f64d40a89e0 2023-11-25T05:03:51.708 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.703+0000 7f64dc790640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:51.708 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.703+0000 7f64dc790640 10 monclient(hunting): authenticate will time out at 1357.946707s 2023-11-25T05:03:51.708 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.704+0000 7f64da52c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f64d40a89e0 0x7f64d413ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:51.708 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.704+0000 7f64d9d2b640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f64d413f460 0x7f64d41421d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33148/0 (socket says 172.21.15.7:33148) 2023-11-25T05:03:51.708 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.704+0000 7f64d9d2b640 1 -- 172.21.15.7:0/1170652337 learned_addr learned my addr 172.21.15.7:0/1170652337 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:03:51.708 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.705+0000 7f64da52c640 10 monclient(hunting): get_auth_request con 0x7f64d40a89e0 auth_method 0 2023-11-25T05:03:51.708 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.705+0000 7f64da52c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:51.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.705+0000 7f64da52c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:51.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.705+0000 7f64da52c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:51.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.705+0000 7f64da52c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:51.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.705+0000 7f64da52c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:51.709 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.705+0000 7f64da52c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:51.710 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.707+0000 7f64d952a640 1 -- 172.21.15.7:0/1170652337 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627086095 0 0) 0x7f64d413f860 con 0x7f64d41428e0 2023-11-25T05:03:51.710 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.707+0000 7f64d952a640 10 monclient(hunting): _negotiate 2023-11-25T05:03:51.710 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.707+0000 7f64d952a640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:51.710 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.707+0000 7f64d952a640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:51.710 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.707+0000 7f64d952a640 10 monclient(hunting): my global_id is 4201 2023-11-25T05:03:51.710 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.707+0000 7f64d952a640 1 -- 172.21.15.7:0/1170652337 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c0003520 con 0x7f64d41428e0 2023-11-25T05:03:51.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.707+0000 7f64da52c640 10 monclient(hunting): handle_auth_done global_id 4169 payload 274 2023-11-25T05:03:51.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.707+0000 7f64da52c640 1 -- 172.21.15.7:0/1170652337 >> v1:172.21.15.90:6789/0 conn(0x7f64d41428e0 legacy=0x7f64d414c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:51.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.707+0000 7f64da52c640 1 -- 172.21.15.7:0/1170652337 >> v1:172.21.15.7:6789/0 conn(0x7f64d413f460 legacy=0x7f64d41421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:51.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.707+0000 7f64da52c640 10 monclient: _finish_hunting 0 2023-11-25T05:03:51.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.707+0000 7f64da52c640 1 monclient: found mon.noname-b 2023-11-25T05:03:51.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.707+0000 7f64da52c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:51.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.707+0000 7f64da52c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:03:51.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.707+0000 7f64da52c640 1 -- 172.21.15.7:0/1170652337 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d40aa9c0 con 0x7f64d40a89e0 2023-11-25T05:03:51.711 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.709+0000 7f64da52c640 1 --2- 172.21.15.7:0/1170652337 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f64d40a89e0 0x7f64d413ef20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f64d00097d0 tx=0x7f64d002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dd166d666293665 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:51.714 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64d952a640 1 -- 172.21.15.7:0/1170652337 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d0031800 con 0x7f64d40a89e0 2023-11-25T05:03:51.714 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64d952a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:51.715 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64d952a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:03:51.715 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64d952a640 10 monclient: dump: 2023-11-25T05:03:51.715 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:03:51.715 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:51.715 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:51.715 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:51.715 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:51.715 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:51.715 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:51.715 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:51.715 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:51.715 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:51.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64d952a640 10 monclient: _finish_auth 0 2023-11-25T05:03:51.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64d952a640 10 monclient: _check_auth_tickets 2023-11-25T05:03:51.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64d952a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:51.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64d952a640 1 -- 172.21.15.7:0/1170652337 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d0031960 con 0x7f64d40a89e0 2023-11-25T05:03:51.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64dc790640 5 monclient: authenticate success, global_id 4169 2023-11-25T05:03:51.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64dc790640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:03:51.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64d952a640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:51.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64d952a640 1 -- 172.21.15.7:0/1170652337 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d0031cf0 con 0x7f64d40a89e0 2023-11-25T05:03:51.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64d952a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:51.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64d952a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:03:51.716 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64d952a640 10 monclient: dump: 2023-11-25T05:03:51.716 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:03:51.716 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:51.717 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:51.717 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:51.717 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:51.717 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:51.717 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:51.717 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:51.717 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:51.717 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:51.717 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64dc790640 10 monclient: get_monmap_and_config success 2023-11-25T05:03:51.717 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.713+0000 7f64dc790640 4 set_mon_vals no callback set 2023-11-25T05:03:51.720 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.714+0000 7f64dc790640 10 monclient: shutdown 2023-11-25T05:03:51.720 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.714+0000 7f64dc790640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:51.720 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.714+0000 7f64dc790640 1 -- 172.21.15.7:0/1170652337 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f64d40a89e0 msgr2=0x7f64d413ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:51.720 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.714+0000 7f64dc790640 1 --2- 172.21.15.7:0/1170652337 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f64d40a89e0 0x7f64d413ef20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f64d00097d0 tx=0x7f64d002b8a0 comp rx=0 tx=0).stop 2023-11-25T05:03:51.721 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.714+0000 7f64dc790640 1 -- 172.21.15.7:0/1170652337 shutdown_connections 2023-11-25T05:03:51.721 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.714+0000 7f64dc790640 1 --2- 172.21.15.7:0/1170652337 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f64d40a89e0 0x7f64d413ef20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:51.721 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.714+0000 7f64dc790640 1 -- 172.21.15.7:0/1170652337 >> 172.21.15.7:0/1170652337 conn(0x7f64d40b7940 msgr2=0x7f64d40b7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:51.721 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.714+0000 7f64dc790640 1 -- 172.21.15.7:0/1170652337 shutdown_connections 2023-11-25T05:03:51.721 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.714+0000 7f64dc790640 1 -- 172.21.15.7:0/1170652337 wait complete. 2023-11-25T05:03:51.721 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.715+0000 7f64dc790640 10 monclient: build_initial_monmap 2023-11-25T05:03:51.721 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.715+0000 7f64dc790640 10 monclient: monmap: 2023-11-25T05:03:51.721 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-11-25T05:03:51.721 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:03:51.721 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2023-11-25T05:03:51.722 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2023-11-25T05:03:51.722 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:51.722 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:51.722 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:03:51.722 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:03:51.722 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:03:51.722 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:51.722 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.715+0000 7f64dc790640 1 librados: starting msgr at 2023-11-25T05:03:51.722 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.715+0000 7f64dc790640 1 librados: starting objecter 2023-11-25T05:03:51.722 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.715+0000 7f64dc790640 1 Processor -- start 2023-11-25T05:03:51.722 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.715+0000 7f64dc790640 1 -- start start 2023-11-25T05:03:51.722 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.715+0000 7f64dc790640 1 librados: setting wanted keys 2023-11-25T05:03:51.722 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.715+0000 7f64dc790640 1 librados: calling monclient init 2023-11-25T05:03:51.723 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.715+0000 7f64dc790640 10 monclient: init 2023-11-25T05:03:51.723 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:51.723 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:03:51.723 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f64d40a89e0 0x7f64d4141390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:51.723 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 10 monclient(hunting): picked mon.noname-c con 0x7f64d40a89e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:51.723 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d413f460 0x7f64d4143270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:51.723 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 10 monclient(hunting): picked mon.noname-b con 0x7f64d413f460 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:51.723 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64d41428e0 0x7f64d4134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:51.723 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 10 monclient(hunting): picked mon.noname-a con 0x7f64d41428e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:51.724 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:51.724 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d4143fb0 con 0x7f64d41428e0 2023-11-25T05:03:51.724 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:51.724 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d41437b0 con 0x7f64d413f460 2023-11-25T05:03:51.724 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:51.725 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d4143a60 con 0x7f64d40a89e0 2023-11-25T05:03:51.725 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:51.725 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.716+0000 7f64dc790640 10 monclient(hunting): authenticate will time out at 1357.959475s 2023-11-25T05:03:51.725 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.717+0000 7f64d9d2b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d413f460 0x7f64d4143270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:51.725 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.717+0000 7f64d9d2b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d413f460 0x7f64d4143270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:40188/0 (socket says 172.21.15.7:40188) 2023-11-25T05:03:51.725 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.717+0000 7f64d9d2b640 1 -- 172.21.15.7:0/3648100359 learned_addr learned my addr 172.21.15.7:0/3648100359 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:03:51.725 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.717+0000 7f64d9d2b640 10 monclient(hunting): get_auth_request con 0x7f64d413f460 auth_method 0 2023-11-25T05:03:51.725 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.717+0000 7f64d9d2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:51.725 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.717+0000 7f64d9d2b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:51.725 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.717+0000 7f64d9d2b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:51.725 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.717+0000 7f64d9d2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:51.725 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.717+0000 7f64d9d2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:51.725 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.717+0000 7f64d9d2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:51.726 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.718+0000 7f64dad2d640 1 --2- 172.21.15.7:0/3648100359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64d41428e0 0x7f64d4134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:51.726 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.719+0000 7f64dad2d640 10 monclient(hunting): get_auth_request con 0x7f64d41428e0 auth_method 0 2023-11-25T05:03:51.726 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.719+0000 7f64dad2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:51.726 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.719+0000 7f64dad2d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:51.726 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.719+0000 7f64dad2d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:51.726 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.719+0000 7f64da52c640 1 --2- 172.21.15.7:0/3648100359 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f64d40a89e0 0x7f64d4141390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:51.727 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:09.755 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.728 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.719+0000 7f64da52c640 10 monclient(hunting): get_auth_request con 0x7f64d40a89e0 auth_method 0 2023-11-25T05:03:51.728 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.719+0000 7f64da52c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:51.728 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.719+0000 7f64da52c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:51.728 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.719+0000 7f64da52c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:51.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.720+0000 7f64d9d2b640 10 monclient(hunting): handle_auth_done global_id 4204 payload 824 2023-11-25T05:03:51.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.720+0000 7f64d9d2b640 1 -- 172.21.15.7:0/3648100359 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f64d40a89e0 msgr2=0x7f64d4141390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:51.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.720+0000 7f64d9d2b640 1 --2- 172.21.15.7:0/3648100359 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f64d40a89e0 0x7f64d4141390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:51.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.720+0000 7f64d9d2b640 1 -- 172.21.15.7:0/3648100359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64d41428e0 msgr2=0x7f64d4134c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:51.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.720+0000 7f64d9d2b640 1 --2- 172.21.15.7:0/3648100359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64d41428e0 0x7f64d4134c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:51.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.720+0000 7f64d9d2b640 10 monclient: _finish_hunting 0 2023-11-25T05:03:51.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.720+0000 7f64d9d2b640 1 monclient: found mon.noname-b 2023-11-25T05:03:51.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.720+0000 7f64d9d2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:51.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.720+0000 7f64d9d2b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:03:51.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.720+0000 7f64d9d2b640 1 -- 172.21.15.7:0/3648100359 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d0009400 con 0x7f64d413f460 2023-11-25T05:03:51.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.722+0000 7f64d9d2b640 1 --2- 172.21.15.7:0/3648100359 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d413f460 0x7f64d4143270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f64c400c4a0 tx=0x7f64c400c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:51.729 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.722+0000 7f64caffd640 10 client.?.objecter ms_handle_connect 0x7f64d413f460 2023-11-25T05:03:51.730 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.722+0000 7f64caffd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:03:51.730 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64caffd640 1 -- 172.21.15.7:0/3648100359 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4005ce0 con 0x7f64d413f460 2023-11-25T05:03:51.730 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:51.730 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64caffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:03:51.730 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64caffd640 10 monclient: dump: 2023-11-25T05:03:51.730 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:03:51.730 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:51.730 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:51.730 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:51.730 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:51.730 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:51.730 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:51.731 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:51.731 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:51.731 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:51.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64caffd640 10 monclient: _finish_auth 0 2023-11-25T05:03:51.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64caffd640 10 monclient: _check_auth_tickets 2023-11-25T05:03:51.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:51.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64caffd640 1 -- 172.21.15.7:0/3648100359 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c4005e40 con 0x7f64d413f460 2023-11-25T05:03:51.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64caffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:51.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64caffd640 1 -- 172.21.15.7:0/3648100359 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4007270 con 0x7f64d413f460 2023-11-25T05:03:51.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64dc790640 5 monclient: authenticate success, global_id 4204 2023-11-25T05:03:51.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64dc790640 10 monclient: _renew_subs 2023-11-25T05:03:51.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64dc790640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:51.731 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64dc790640 1 -- 172.21.15.7:0/3648100359 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d40b7390 con 0x7f64d413f460 2023-11-25T05:03:51.732 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64d8d29640 4 set_mon_vals no callback set 2023-11-25T05:03:51.732 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:51.732 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:03:51.732 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64caffd640 10 monclient: dump: 2023-11-25T05:03:51.732 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:03:51.732 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:51.732 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:51.732 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:51.732 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:51.732 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:51.732 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:51.732 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:51.732 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:51.733 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:51.733 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64dc790640 10 client.4204.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:03:51.733 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64dc790640 10 monclient: _renew_subs 2023-11-25T05:03:51.733 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64dc790640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:51.733 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64dc790640 1 -- 172.21.15.7:0/3648100359 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d40b76c0 con 0x7f64d413f460 2023-11-25T05:03:51.733 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.725+0000 7f64dc790640 1 librados: init done 2023-11-25T05:03:51.733 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.726+0000 7f64dc790640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:51.733 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.726+0000 7f64dc790640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:51.733 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.726+0000 7f64dc790640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:51.733 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.726+0000 7f64dc790640 1 -- 172.21.15.7:0/3648100359 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d4069c00 con 0x7f64d413f460 2023-11-25T05:03:51.745 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.744+0000 7f64caffd640 1 -- 172.21.15.7:0/3648100359 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f64c4007490 con 0x7f64d413f460 2023-11-25T05:03:51.746 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.744+0000 7f64caffd640 1 --2- 172.21.15.7:0/3648100359 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f64b806c790 0x7f64b806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:51.746 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.744+0000 7f64da52c640 1 --2- 172.21.15.7:0/3648100359 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f64b806c790 0x7f64b806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:51.746 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.744+0000 7f64da52c640 10 monclient: get_auth_request con 0x7f64b806c790 auth_method 0 2023-11-25T05:03:51.746 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.745+0000 7f64caffd640 10 client.4204.objecter ms_handle_connect 0x7f64b806c790 2023-11-25T05:03:51.746 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.745+0000 7f64da52c640 1 --2- 172.21.15.7:0/3648100359 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f64b806c790 0x7f64b806ec50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f64d0036000 tx=0x7f64d003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:51.748 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.746+0000 7f64caffd640 1 -- 172.21.15.7:0/3648100359 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f64c4057d90 con 0x7f64d413f460 2023-11-25T05:03:51.748 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.746+0000 7f64caffd640 10 client.4204.objecter ms_dispatch 0x7f64d4000b90 osd_map(10..10 src has 1..10) v4 2023-11-25T05:03:51.748 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.746+0000 7f64caffd640 3 client.4204.objecter handle_osd_map got epochs [10,10] > 0 2023-11-25T05:03:51.748 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.746+0000 7f64caffd640 3 client.4204.objecter handle_osd_map decoding full epoch 10 2023-11-25T05:03:51.748 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.746+0000 7f64caffd640 20 client.4204.objecter dump_active .. 0 homeless 2023-11-25T05:03:51.810 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:09.860 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.834 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:09.919 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.924 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:09.968 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.928 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:10.005 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.933 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:10.019 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.968 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:10.018 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:51.985 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.983+0000 7f64caffd640 1 -- 172.21.15.7:0/3648100359 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64c40540d0 con 0x7f64d413f460 2023-11-25T05:03:51.985 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.983+0000 7f64caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:51.985 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:51.983+0000 7f64caffd640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:03:51.990 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:10.022 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.005 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:10.093 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.088 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:10.139 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.162 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.161+0000 7f64dc790640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:03:52.162 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.161+0000 7f64dc790640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:03:52.162 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.161+0000 7f64dc790640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:03:52.163 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.161+0000 7f64dc790640 1 -- 172.21.15.7:0/3648100359 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64d413dda0 con 0x7f64d413f460 2023-11-25T05:03:52.176 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.175+0000 7f64caffd640 1 -- 172.21.15.7:0/3648100359 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f64c40576f0 con 0x7f64d413f460 2023-11-25T05:03:52.237 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.175+0000 7f64caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:03:52.237 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.175+0000 7f64caffd640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:03:52.238 INFO:teuthology.misc.health.smithi007.stdout: 2023-11-25T05:03:52.238 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"59cf1eff-816d-4eda-b513-a2de7ea218f7","created":"2023-11-25T05:03:35.598582+0000","modified":"2023-11-25T05:03:41.565581+0000","last_up_change":"0.000000","last_in_change":"2023-11-25T05:03:41.565581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c87807e-7b82-4866-ba36-60b44a25f317","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0e252ab8-9524-4ccc-b08f-d95bbf05fbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccda96b8-aa9e-4c99-8d84-bba19ca0483b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e4b8700a-2e28-4e69-8577-465666b1b051","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"36c88a76-0467-4e37-85b4-dfd5953855a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ebf452a0-1b17-488a-bbea-2e953c932ac7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5aaf35c3-5c06-45e5-b788-8d3207c47cd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d50dd23a-f864-4593-ac00-4d8afed21dd0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-25T05:03:52.240 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 10 librados: watch_flush enter 2023-11-25T05:03:52.240 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 10 librados: watch_flush exit 2023-11-25T05:03:52.240 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 20 client.4204.objecter shutdown clearing up homeless session... 2023-11-25T05:03:52.240 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 10 client.4204.objecter successfully canceled tick 2023-11-25T05:03:52.240 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 1 -- 172.21.15.7:0/3648100359 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f64b806c790 msgr2=0x7f64b806ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:52.241 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 1 --2- 172.21.15.7:0/3648100359 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f64b806c790 0x7f64b806ec50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f64d0036000 tx=0x7f64d003c040 comp rx=0 tx=0).stop 2023-11-25T05:03:52.241 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 10 monclient: shutdown 2023-11-25T05:03:52.241 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:52.241 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 1 -- 172.21.15.7:0/3648100359 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d413f460 msgr2=0x7f64d4143270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:52.241 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 1 --2- 172.21.15.7:0/3648100359 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d413f460 0x7f64d4143270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f64c400c4a0 tx=0x7f64c400c960 comp rx=0 tx=0).stop 2023-11-25T05:03:52.241 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 1 -- 172.21.15.7:0/3648100359 shutdown_connections 2023-11-25T05:03:52.241 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 1 --2- 172.21.15.7:0/3648100359 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f64b806c790 0x7f64b806ec50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:52.241 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 1 --2- 172.21.15.7:0/3648100359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64d41428e0 0x7f64d4134c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:52.241 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 1 --2- 172.21.15.7:0/3648100359 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d413f460 0x7f64d4143270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:52.242 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 1 --2- 172.21.15.7:0/3648100359 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f64d40a89e0 0x7f64d4141390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:52.242 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 1 -- 172.21.15.7:0/3648100359 >> 172.21.15.7:0/3648100359 conn(0x7f64d40b7940 msgr2=0x7f64d413d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:52.242 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 1 -- 172.21.15.7:0/3648100359 shutdown_connections 2023-11-25T05:03:52.242 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 1 -- 172.21.15.7:0/3648100359 wait complete. 2023-11-25T05:03:52.242 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:52.177+0000 7f64dc790640 1 librados: shutdown 2023-11-25T05:03:52.242 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-25T05:03:52.243 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:10.301 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.243 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:10.248 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.247 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:10.297 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.275 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:10.308 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.302 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:10.378 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.302 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:10.387 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.349 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:10.434 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.380 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:10.467 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.381 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:10.432 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.490 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:10.534 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.509 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:10.595 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.558 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:10.608 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.568 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:10.601 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.660 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:10.745 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.661 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:10.737 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.672 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:10.723 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.748 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:10.836 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.777 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:10.822 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.800 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:10.885 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.847 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:10.898 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.867 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:10.900 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.952 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:11.037 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.961 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:11.037 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:52.964 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:11.015 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.042 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:11.129 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.071 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:11.115 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.093 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:11.178 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.153 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:11.203 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.158 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:11.191 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.244 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:11.328 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.252 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:11.301 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.256 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:11.332 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.307 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:11.394 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.335 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:11.379 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.361 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:11.446 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.417 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:11.467 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.427 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:11.459 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.509 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:11.594 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.520 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:11.570 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.523 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:11.599 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.579 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:11.667 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.599 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:11.643 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.626 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:11.711 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.682 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:11.732 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.780 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:11.865 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.796 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:11.872 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.844 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:11.931 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.898 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:11.930 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.899 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:11.984 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.952 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:11.985 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:53.997 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:12.047 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.055 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:12.101 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.067 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:12.143 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.075 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:12.119 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.120 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:12.207 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.130 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:12.174 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.158 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:12.208 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.212 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:12.262 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.278 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:12.363 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.334 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:12.419 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.336 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:12.412 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.390 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:12.475 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.401 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:12.488 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.446 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:12.532 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.858 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:12.934 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.899 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:12.987 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.914 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:12.990 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:54.956 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:13.044 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:59.049 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-25T05:03:59.053 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:17.086 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:59.142 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.140+0000 7fe5e7590640 10 monclient: get_monmap_and_config 2023-11-25T05:03:59.142 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.140+0000 7fe5e7590640 10 monclient: build_initial_monmap 2023-11-25T05:03:59.142 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.140+0000 7fe5e7590640 10 monclient: monmap: 2023-11-25T05:03:59.142 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-11-25T05:03:59.142 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:59.142 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:03:59.141617+0000 2023-11-25T05:03:59.142 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:03:59.141617+0000 2023-11-25T05:03:59.142 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:59.142 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:59.143 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:03:59.143 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:03:59.143 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:03:59.143 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:03:59.143 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:03:59.143 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:59.144 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.141+0000 7fe5e7590640 1 Processor -- start 2023-11-25T05:03:59.144 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.141+0000 7fe5e7590640 1 -- start start 2023-11-25T05:03:59.144 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.141+0000 7fe5e7590640 10 monclient: init 2023-11-25T05:03:59.144 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.142+0000 7fe5e7590640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:59.144 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.142+0000 7fe5e7590640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2023-11-25T05:03:59.144 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.142+0000 7fe5e7590640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5e00a8a00 addr v1:172.21.15.90:6789/0 2023-11-25T05:03:59.144 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.142+0000 7fe5e7590640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5e0146150 0x7fe5e00a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:59.144 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.142+0000 7fe5e7590640 10 monclient(hunting): picked mon.noname-b con 0x7fe5e0146150 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:59.144 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.142+0000 7fe5e7590640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe5e013eca0 addr v1:172.21.15.7:6789/0 2023-11-25T05:03:59.144 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.142+0000 7fe5e7590640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e01455b0 con 0x7fe5e013eca0 2023-11-25T05:03:59.145 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.142+0000 7fe5e7590640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e00aa970 con 0x7fe5e00a8a00 2023-11-25T05:03:59.145 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.142+0000 7fe5e7590640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:59.145 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.142+0000 7fe5e7590640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e00aac50 con 0x7fe5e0146150 2023-11-25T05:03:59.145 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.142+0000 7fe5e7590640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:59.145 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.142+0000 7fe5e7590640 10 monclient(hunting): authenticate will time out at 1365.385466s 2023-11-25T05:03:59.145 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.143+0000 7fe5e4b2b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5e0146150 0x7fe5e00a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:59.145 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.143+0000 7fe5e5b2d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe5e013eca0 0x7fe5e013f130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35834/0 (socket says 172.21.15.7:35834) 2023-11-25T05:03:59.146 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.143+0000 7fe5e5b2d640 1 -- 172.21.15.7:0/3668787205 learned_addr learned my addr 172.21.15.7:0/3668787205 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:03:59.146 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.144+0000 7fe5e4b2b640 10 monclient(hunting): get_auth_request con 0x7fe5e0146150 auth_method 0 2023-11-25T05:03:59.146 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.144+0000 7fe5e4b2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:59.146 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.144+0000 7fe5e4b2b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:59.146 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.144+0000 7fe5e4b2b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:59.146 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.144+0000 7fe5e4b2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:59.146 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.144+0000 7fe5e4b2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:59.146 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.144+0000 7fe5e4b2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:59.148 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.146+0000 7fe5d7fff640 1 -- 172.21.15.7:0/3668787205 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721474446 0 0) 0x7fe5e01455b0 con 0x7fe5e013eca0 2023-11-25T05:03:59.148 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.146+0000 7fe5d7fff640 10 monclient(hunting): _negotiate 2023-11-25T05:03:59.148 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.146+0000 7fe5d7fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:59.148 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.146+0000 7fe5d7fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:59.148 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.146+0000 7fe5d7fff640 10 monclient(hunting): my global_id is 4194 2023-11-25T05:03:59.148 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.146+0000 7fe5d7fff640 1 -- 172.21.15.7:0/3668787205 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5cc003520 con 0x7fe5e013eca0 2023-11-25T05:03:59.148 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.147+0000 7fe5e4b2b640 10 monclient(hunting): handle_auth_done global_id 4175 payload 274 2023-11-25T05:03:59.148 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.147+0000 7fe5e4b2b640 1 -- 172.21.15.7:0/3668787205 >> v1:172.21.15.90:6789/0 conn(0x7fe5e00a8a00 legacy=0x7fe5e0145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:59.149 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.147+0000 7fe5e4b2b640 1 -- 172.21.15.7:0/3668787205 >> v1:172.21.15.7:6789/0 conn(0x7fe5e013eca0 legacy=0x7fe5e013f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:59.149 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.147+0000 7fe5e4b2b640 10 monclient: _finish_hunting 0 2023-11-25T05:03:59.149 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.147+0000 7fe5e4b2b640 1 monclient: found mon.noname-b 2023-11-25T05:03:59.149 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.147+0000 7fe5e4b2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:59.149 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.147+0000 7fe5e4b2b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:03:59.149 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.147+0000 7fe5e4b2b640 1 -- 172.21.15.7:0/3668787205 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e00aaf30 con 0x7fe5e0146150 2023-11-25T05:03:59.151 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.150+0000 7fe5e4b2b640 1 --2- 172.21.15.7:0/3668787205 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5e0146150 0x7fe5e00a8de0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe5d00097d0 tx=0x7fe5d002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4d09ef185a75390 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:59.162 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5d7fff640 1 -- 172.21.15.7:0/3668787205 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0030800 con 0x7fe5e0146150 2023-11-25T05:03:59.162 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:59.162 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5d7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:03:59.162 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5d7fff640 10 monclient: dump: 2023-11-25T05:03:59.162 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:03:59.162 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:59.162 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:59.162 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:59.163 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:59.163 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:59.163 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:59.163 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:59.163 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:59.163 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:59.163 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5d7fff640 10 monclient: _finish_auth 0 2023-11-25T05:03:59.163 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5d7fff640 10 monclient: _check_auth_tickets 2023-11-25T05:03:59.163 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:59.163 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5d7fff640 1 -- 172.21.15.7:0/3668787205 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d0030960 con 0x7fe5e0146150 2023-11-25T05:03:59.163 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5e7590640 5 monclient: authenticate success, global_id 4175 2023-11-25T05:03:59.164 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5e7590640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:03:59.164 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5d7fff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:59.164 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5d7fff640 1 -- 172.21.15.7:0/3668787205 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0030cf0 con 0x7fe5e0146150 2023-11-25T05:03:59.164 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:59.164 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5d7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:03:59.164 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5d7fff640 10 monclient: dump: 2023-11-25T05:03:59.164 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:03:59.164 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:59.164 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:59.164 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:59.164 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:59.164 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:59.164 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:59.165 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:59.165 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:59.165 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:59.165 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5e7590640 10 monclient: get_monmap_and_config success 2023-11-25T05:03:59.165 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.154+0000 7fe5e7590640 4 set_mon_vals no callback set 2023-11-25T05:03:59.165 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.155+0000 7fe5e7590640 10 monclient: shutdown 2023-11-25T05:03:59.165 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.155+0000 7fe5e7590640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:59.165 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.155+0000 7fe5e7590640 1 -- 172.21.15.7:0/3668787205 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5e0146150 msgr2=0x7fe5e00a8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:59.165 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.155+0000 7fe5e7590640 1 --2- 172.21.15.7:0/3668787205 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5e0146150 0x7fe5e00a8de0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe5d00097d0 tx=0x7fe5d002b940 comp rx=0 tx=0).stop 2023-11-25T05:03:59.165 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.155+0000 7fe5e7590640 1 -- 172.21.15.7:0/3668787205 shutdown_connections 2023-11-25T05:03:59.165 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.155+0000 7fe5e7590640 1 --2- 172.21.15.7:0/3668787205 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5e0146150 0x7fe5e00a8de0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:59.165 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.155+0000 7fe5e7590640 1 -- 172.21.15.7:0/3668787205 >> 172.21.15.7:0/3668787205 conn(0x7fe5e0066e80 msgr2=0x7fe5e00b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:59.165 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.156+0000 7fe5e7590640 1 -- 172.21.15.7:0/3668787205 shutdown_connections 2023-11-25T05:03:59.166 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.156+0000 7fe5e7590640 1 -- 172.21.15.7:0/3668787205 wait complete. 2023-11-25T05:03:59.166 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.157+0000 7fe5e7590640 10 monclient: build_initial_monmap 2023-11-25T05:03:59.166 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.157+0000 7fe5e7590640 10 monclient: monmap: 2023-11-25T05:03:59.166 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-11-25T05:03:59.166 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:03:59.166 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2023-11-25T05:03:59.166 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2023-11-25T05:03:59.166 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:03:59.166 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:59.166 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:03:59.166 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:03:59.166 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:03:59.166 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:59.167 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.157+0000 7fe5e7590640 1 librados: starting msgr at 2023-11-25T05:03:59.167 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.157+0000 7fe5e7590640 1 librados: starting objecter 2023-11-25T05:03:59.167 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.157+0000 7fe5e7590640 1 Processor -- start 2023-11-25T05:03:59.167 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.157+0000 7fe5e7590640 1 -- start start 2023-11-25T05:03:59.167 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.157+0000 7fe5e7590640 1 librados: setting wanted keys 2023-11-25T05:03:59.167 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.157+0000 7fe5e7590640 1 librados: calling monclient init 2023-11-25T05:03:59.167 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.157+0000 7fe5e7590640 10 monclient: init 2023-11-25T05:03:59.167 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 10 monclient: _reopen_session rank -1 2023-11-25T05:03:59.167 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-25T05:03:59.167 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e00a8a00 0x7fe5e0142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:59.167 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 10 monclient(hunting): picked mon.noname-a con 0x7fe5e00a8a00 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:03:59.167 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5e013eca0 0x7fe5e01da460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:59.168 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 10 monclient(hunting): picked mon.noname-b con 0x7fe5e013eca0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:03:59.168 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5e0146150 0x7fe5e01da9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:59.168 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 10 monclient(hunting): picked mon.noname-c con 0x7fe5e0146150 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:03:59.168 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:59.168 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e0138dc0 con 0x7fe5e00a8a00 2023-11-25T05:03:59.168 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:59.168 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e01daee0 con 0x7fe5e013eca0 2023-11-25T05:03:59.168 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 10 monclient(hunting): start opening mon connection 2023-11-25T05:03:59.168 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e01db050 con 0x7fe5e0146150 2023-11-25T05:03:59.168 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 10 monclient(hunting): _renew_subs 2023-11-25T05:03:59.168 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.158+0000 7fe5e7590640 10 monclient(hunting): authenticate will time out at 1365.401464s 2023-11-25T05:03:59.168 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.159+0000 7fe5e4b2b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5e013eca0 0x7fe5e01da460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:59.168 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.159+0000 7fe5e5b2d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5e0146150 0x7fe5e01da9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:59.169 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.159+0000 7fe5e532c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e00a8a00 0x7fe5e0142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:59.169 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.159+0000 7fe5e532c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e00a8a00 0x7fe5e0142cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36392/0 (socket says 172.21.15.7:36392) 2023-11-25T05:03:59.169 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.159+0000 7fe5e532c640 1 -- 172.21.15.7:0/3934672050 learned_addr learned my addr 172.21.15.7:0/3934672050 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:03:59.169 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.159+0000 7fe5e532c640 10 monclient(hunting): get_auth_request con 0x7fe5e00a8a00 auth_method 0 2023-11-25T05:03:59.169 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.159+0000 7fe5e532c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:59.169 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.159+0000 7fe5e532c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:59.169 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.159+0000 7fe5e532c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:59.169 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.159+0000 7fe5e4b2b640 10 monclient(hunting): get_auth_request con 0x7fe5e013eca0 auth_method 0 2023-11-25T05:03:59.169 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e4b2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:59.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e4b2b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:59.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e4b2b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:59.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e5b2d640 10 monclient(hunting): get_auth_request con 0x7fe5e0146150 auth_method 0 2023-11-25T05:03:59.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e5b2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:03:59.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e5b2d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:03:59.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e5b2d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:03:59.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e532c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:59.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e532c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:59.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e532c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:59.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e5b2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:59.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e5b2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:59.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e5b2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:59.171 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e4b2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:03:59.171 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e4b2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:03:59.171 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.160+0000 7fe5e4b2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:03:59.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.163+0000 7fe5e532c640 10 monclient(hunting): handle_auth_done global_id 4197 payload 824 2023-11-25T05:03:59.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.163+0000 7fe5e532c640 1 -- 172.21.15.7:0/3934672050 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5e0146150 msgr2=0x7fe5e01da9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:59.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.163+0000 7fe5e532c640 1 --2- 172.21.15.7:0/3934672050 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5e0146150 0x7fe5e01da9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:59.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.163+0000 7fe5e532c640 1 -- 172.21.15.7:0/3934672050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5e013eca0 msgr2=0x7fe5e01da460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:59.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.163+0000 7fe5e532c640 1 --2- 172.21.15.7:0/3934672050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5e013eca0 0x7fe5e01da460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:59.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.163+0000 7fe5e532c640 10 monclient: _finish_hunting 0 2023-11-25T05:03:59.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.163+0000 7fe5e532c640 1 monclient: found mon.noname-a 2023-11-25T05:03:59.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.163+0000 7fe5e532c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:03:59.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.163+0000 7fe5e532c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:03:59.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.163+0000 7fe5e532c640 1 -- 172.21.15.7:0/3934672050 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d0009400 con 0x7fe5e00a8a00 2023-11-25T05:03:59.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.166+0000 7fe5d5ffb640 10 client.?.objecter ms_handle_connect 0x7fe5e00a8a00 2023-11-25T05:03:59.173 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.166+0000 7fe5d5ffb640 10 client.?.objecter resend_mon_ops 2023-11-25T05:03:59.173 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.166+0000 7fe5e532c640 1 --2- 172.21.15.7:0/3934672050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e00a8a00 0x7fe5e0142cb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc00c450 tx=0x7fe5dc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:59.173 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5d5ffb640 1 -- 172.21.15.7:0/3934672050 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc0052d0 con 0x7fe5e00a8a00 2023-11-25T05:03:59.173 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:59.173 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5d5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:03:59.173 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5d5ffb640 10 monclient: dump: 2023-11-25T05:03:59.173 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:03:59.173 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:59.173 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:59.173 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:59.173 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:59.173 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:59.173 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:59.173 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:59.174 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:59.174 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:59.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5d5ffb640 10 monclient: _finish_auth 0 2023-11-25T05:03:59.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5d5ffb640 10 monclient: _check_auth_tickets 2023-11-25T05:03:59.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:03:59.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5e7590640 5 monclient: authenticate success, global_id 4197 2023-11-25T05:03:59.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5e7590640 10 monclient: _renew_subs 2023-11-25T05:03:59.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5e7590640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:03:59.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5d5ffb640 1 -- 172.21.15.7:0/3934672050 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5dc005430 con 0x7fe5e00a8a00 2023-11-25T05:03:59.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5e7590640 1 -- 172.21.15.7:0/3934672050 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e0139ab0 con 0x7fe5e00a8a00 2023-11-25T05:03:59.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5d5ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:03:59.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5d5ffb640 1 -- 172.21.15.7:0/3934672050 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc0052d0 con 0x7fe5e00a8a00 2023-11-25T05:03:59.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:03:59.175 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:03:59.175 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5d5ffb640 10 monclient: dump: 2023-11-25T05:03:59.175 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:03:59.175 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:03:59.175 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:59.175 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:03:59.175 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:03:59.175 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:03:59.175 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:03:59.175 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:03:59.175 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:03:59.175 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:03:59.175 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5e7590640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:03:59.176 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5e7590640 10 monclient: _renew_subs 2023-11-25T05:03:59.176 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5e7590640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:03:59.176 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5e7590640 1 -- 172.21.15.7:0/3934672050 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e01d6ed0 con 0x7fe5e00a8a00 2023-11-25T05:03:59.176 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5e7590640 1 librados: init done 2023-11-25T05:03:59.176 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.170+0000 7fe5d77fe640 4 set_mon_vals no callback set 2023-11-25T05:03:59.176 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.172+0000 7fe5e7590640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:59.177 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.172+0000 7fe5e7590640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:59.177 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.172+0000 7fe5e7590640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:03:59.177 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.172+0000 7fe5e7590640 1 -- 172.21.15.7:0/3934672050 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e0139db0 con 0x7fe5e00a8a00 2023-11-25T05:03:59.196 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.194+0000 7fe5d5ffb640 1 -- 172.21.15.7:0/3934672050 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7fe5dc012630 con 0x7fe5e00a8a00 2023-11-25T05:03:59.196 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.195+0000 7fe5d5ffb640 1 --2- 172.21.15.7:0/3934672050 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe5c406c7a0 0x7fe5c406ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:03:59.208 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.207+0000 7fe5e4b2b640 1 --2- 172.21.15.7:0/3934672050 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe5c406c7a0 0x7fe5c406ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:03:59.211 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.210+0000 7fe5e4b2b640 10 monclient: get_auth_request con 0x7fe5c406c7a0 auth_method 0 2023-11-25T05:03:59.211 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.210+0000 7fe5e4b2b640 1 --2- 172.21.15.7:0/3934672050 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe5c406c7a0 0x7fe5c406ec60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0036000 tx=0x7fe5d003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:03:59.211 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.210+0000 7fe5d5ffb640 10 client.4197.objecter ms_handle_connect 0x7fe5c406c7a0 2023-11-25T05:03:59.301 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.300+0000 7fe5d5ffb640 1 -- 172.21.15.7:0/3934672050 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fe5dc058500 con 0x7fe5e00a8a00 2023-11-25T05:03:59.301 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.300+0000 7fe5d5ffb640 10 client.4197.objecter ms_dispatch 0x7fe5e0000b90 osd_map(10..10 src has 1..10) v4 2023-11-25T05:03:59.301 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.300+0000 7fe5d5ffb640 3 client.4197.objecter handle_osd_map got epochs [10,10] > 0 2023-11-25T05:03:59.302 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.300+0000 7fe5d5ffb640 3 client.4197.objecter handle_osd_map decoding full epoch 10 2023-11-25T05:03:59.302 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.300+0000 7fe5d5ffb640 20 client.4197.objecter dump_active .. 0 homeless 2023-11-25T05:03:59.355 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:17.406 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:59.575 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.573+0000 7fe5d5ffb640 1 -- 172.21.15.7:0/3934672050 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5dc054840 con 0x7fe5e00a8a00 2023-11-25T05:03:59.575 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.573+0000 7fe5d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:03:59.575 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.573+0000 7fe5d5ffb640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:03:59.659 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:17.703 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:59.726 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:17.776 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:03:59.845 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.844+0000 7fe5e7590640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:03:59.845 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.844+0000 7fe5e7590640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:03:59.846 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.844+0000 7fe5e7590640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:03:59.846 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.844+0000 7fe5e7590640 1 -- 172.21.15.7:0/3934672050 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5e0069c00 con 0x7fe5e00a8a00 2023-11-25T05:03:59.863 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.862+0000 7fe5d5ffb640 1 -- 172.21.15.7:0/3934672050 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fe5dc057e60 con 0x7fe5e00a8a00 2023-11-25T05:03:59.863 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.862+0000 7fe5d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:03:59.864 INFO:teuthology.misc.health.smithi007.stdout: 2023-11-25T05:03:59.864 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"59cf1eff-816d-4eda-b513-a2de7ea218f7","created":"2023-11-25T05:03:35.598582+0000","modified":"2023-11-25T05:03:41.565581+0000","last_up_change":"0.000000","last_in_change":"2023-11-25T05:03:41.565581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c87807e-7b82-4866-ba36-60b44a25f317","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0e252ab8-9524-4ccc-b08f-d95bbf05fbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccda96b8-aa9e-4c99-8d84-bba19ca0483b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e4b8700a-2e28-4e69-8577-465666b1b051","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"36c88a76-0467-4e37-85b4-dfd5953855a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ebf452a0-1b17-488a-bbea-2e953c932ac7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5aaf35c3-5c06-45e5-b788-8d3207c47cd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d50dd23a-f864-4593-ac00-4d8afed21dd0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-25T05:03:59.864 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.862+0000 7fe5d5ffb640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:03:59.865 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 10 librados: watch_flush enter 2023-11-25T05:03:59.866 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 10 librados: watch_flush exit 2023-11-25T05:03:59.866 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 20 client.4197.objecter shutdown clearing up homeless session... 2023-11-25T05:03:59.866 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 10 client.4197.objecter successfully canceled tick 2023-11-25T05:03:59.866 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 1 -- 172.21.15.7:0/3934672050 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe5c406c7a0 msgr2=0x7fe5c406ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:59.866 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 1 --2- 172.21.15.7:0/3934672050 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe5c406c7a0 0x7fe5c406ec60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0036000 tx=0x7fe5d003b040 comp rx=0 tx=0).stop 2023-11-25T05:03:59.866 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 10 monclient: shutdown 2023-11-25T05:03:59.866 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:03:59.866 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 1 -- 172.21.15.7:0/3934672050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e00a8a00 msgr2=0x7fe5e0142cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:03:59.866 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 1 --2- 172.21.15.7:0/3934672050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e00a8a00 0x7fe5e0142cb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc00c450 tx=0x7fe5dc00c910 comp rx=0 tx=0).stop 2023-11-25T05:03:59.867 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 1 -- 172.21.15.7:0/3934672050 shutdown_connections 2023-11-25T05:03:59.867 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 1 --2- 172.21.15.7:0/3934672050 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe5c406c7a0 0x7fe5c406ec60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:59.867 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 1 --2- 172.21.15.7:0/3934672050 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5e0146150 0x7fe5e01da9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:59.867 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 1 --2- 172.21.15.7:0/3934672050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5e013eca0 0x7fe5e01da460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:59.867 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 1 --2- 172.21.15.7:0/3934672050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5e00a8a00 0x7fe5e0142cb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:03:59.867 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.864+0000 7fe5e7590640 1 -- 172.21.15.7:0/3934672050 >> 172.21.15.7:0/3934672050 conn(0x7fe5e0066e80 msgr2=0x7fe5e013cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:03:59.868 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.865+0000 7fe5e7590640 1 -- 172.21.15.7:0/3934672050 shutdown_connections 2023-11-25T05:03:59.868 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.865+0000 7fe5e7590640 1 -- 172.21.15.7:0/3934672050 wait complete. 2023-11-25T05:03:59.868 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:03:59.865+0000 7fe5e7590640 1 librados: shutdown 2023-11-25T05:03:59.880 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-25T05:04:00.131 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:18.216 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:00.953 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:19.040 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:00.957 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:19.043 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:01.066 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:19.141 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:04.312 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:22.344 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:04.315 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:22.348 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:04.630 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:22.680 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:04.633 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:22.684 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:04.929 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:22.973 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:04.935 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:22.979 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:05.176 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:23.226 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:05.180 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:23.230 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:05.797 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:23.882 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:05.804 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:23.889 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:06.245 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:24.333 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:06.252 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:24.340 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:06.687 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-25T05:04:06.699 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:24.783 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:06.702 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:24.787 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:06.791 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.790+0000 7f1040d33640 10 monclient: get_monmap_and_config 2023-11-25T05:04:06.792 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.790+0000 7f1040d33640 10 monclient: build_initial_monmap 2023-11-25T05:04:06.792 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.790+0000 7f1040d33640 10 monclient: monmap: 2023-11-25T05:04:06.792 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-11-25T05:04:06.792 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:06.792 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:04:06.791477+0000 2023-11-25T05:04:06.792 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:04:06.791477+0000 2023-11-25T05:04:06.792 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:06.792 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:06.792 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:06.793 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:06.793 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:06.793 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:06.793 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:06.793 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:06.795 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 1 Processor -- start 2023-11-25T05:04:06.795 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 1 -- start start 2023-11-25T05:04:06.795 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 10 monclient: init 2023-11-25T05:04:06.795 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:06.795 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2023-11-25T05:04:06.795 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f103c0a89e0 0x7f103c13ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:06.795 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 10 monclient(hunting): picked mon.noname-b con 0x7f103c0a89e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:06.795 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f103c13f460 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:06.796 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f103c1428e0 0x7f103c14c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:06.796 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 10 monclient(hunting): picked mon.noname-a con 0x7f103c1428e0 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:06.796 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c0aa770 con 0x7f103c13f460 2023-11-25T05:04:06.796 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:06.796 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f103c13f860 con 0x7f103c1428e0 2023-11-25T05:04:06.796 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:06.796 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f103c142d70 con 0x7f103c0a89e0 2023-11-25T05:04:06.796 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.791+0000 7f1040d33640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:06.796 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.792+0000 7f1040d33640 10 monclient(hunting): authenticate will time out at 1373.034829s 2023-11-25T05:04:06.796 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.792+0000 7f103a59c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f103c0a89e0 0x7f103c13ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:06.796 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.792+0000 7f103ad9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f103c1428e0 0x7f103c14c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:06.797 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.793+0000 7f103a59c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f103c0a89e0 0x7f103c13ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:42146/0 (socket says 172.21.15.7:42146) 2023-11-25T05:04:06.797 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.793+0000 7f103a59c640 1 -- 172.21.15.7:0/2707658037 learned_addr learned my addr 172.21.15.7:0/2707658037 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:06.797 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.793+0000 7f103a59c640 10 monclient(hunting): get_auth_request con 0x7f103c0a89e0 auth_method 0 2023-11-25T05:04:06.797 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.793+0000 7f103a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:06.797 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.793+0000 7f103a59c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:06.798 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.793+0000 7f103a59c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:06.798 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.793+0000 7f103ad9d640 10 monclient(hunting): get_auth_request con 0x7f103c1428e0 auth_method 0 2023-11-25T05:04:06.798 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.793+0000 7f103ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:06.798 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.793+0000 7f103ad9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:06.798 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.793+0000 7f103ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:06.798 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.793+0000 7f103a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:06.798 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.793+0000 7f103a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:06.798 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.793+0000 7f103a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:06.799 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.794+0000 7f103ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:06.800 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.794+0000 7f103ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:06.800 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.794+0000 7f103ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:06.800 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.796+0000 7f103a59c640 10 monclient(hunting): handle_auth_done global_id 4181 payload 274 2023-11-25T05:04:06.800 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.796+0000 7f103a59c640 1 -- 172.21.15.7:0/2707658037 >> v2:172.21.15.7:3300/0 conn(0x7f103c1428e0 msgr2=0x7f103c14c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:06.800 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.796+0000 7f103a59c640 1 --2- 172.21.15.7:0/2707658037 >> v2:172.21.15.7:3300/0 conn(0x7f103c1428e0 0x7f103c14c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:06.800 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.796+0000 7f103a59c640 1 -- 172.21.15.7:0/2707658037 >> v1:172.21.15.7:6789/0 conn(0x7f103c13f460 legacy=0x7f103c1421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:06.800 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.796+0000 7f103a59c640 10 monclient: _finish_hunting 0 2023-11-25T05:04:06.800 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.796+0000 7f103a59c640 1 monclient: found mon.noname-b 2023-11-25T05:04:06.800 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.796+0000 7f103a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:06.800 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.796+0000 7f103a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:04:06.800 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.796+0000 7f103a59c640 1 -- 172.21.15.7:0/2707658037 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c0aa9c0 con 0x7f103c0a89e0 2023-11-25T05:04:06.801 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.796+0000 7f103ad9d640 1 --2- 172.21.15.7:0/2707658037 >> v2:172.21.15.7:3300/0 conn(0x7f103c1428e0 0x7f103c14c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-25T05:04:06.801 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.798+0000 7f103a59c640 1 --2- 172.21.15.7:0/2707658037 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f103c0a89e0 0x7f103c13ef20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f102c009870 tx=0x7f102c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c068307d6ef150fb server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:06.805 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f103959a640 1 -- 172.21.15.7:0/2707658037 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c031640 con 0x7f103c0a89e0 2023-11-25T05:04:06.805 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f103959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:06.806 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f103959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:06.806 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f103959a640 10 monclient: dump: 2023-11-25T05:04:06.806 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:06.806 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:06.806 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:06.806 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:06.806 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:06.806 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:06.806 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:06.806 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:06.806 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:06.807 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:06.807 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f103959a640 10 monclient: _finish_auth 0 2023-11-25T05:04:06.807 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f103959a640 10 monclient: _check_auth_tickets 2023-11-25T05:04:06.807 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f103959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:06.807 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f103959a640 1 -- 172.21.15.7:0/2707658037 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102c0317a0 con 0x7f103c0a89e0 2023-11-25T05:04:06.807 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f1040d33640 5 monclient: authenticate success, global_id 4181 2023-11-25T05:04:06.807 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f1040d33640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:06.807 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f103959a640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:06.807 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f1040d33640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:06.807 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f1040d33640 4 set_mon_vals no callback set 2023-11-25T05:04:06.807 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f103959a640 1 -- 172.21.15.7:0/2707658037 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c031b30 con 0x7f103c0a89e0 2023-11-25T05:04:06.807 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f103959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:06.807 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f103959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:06.808 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.803+0000 7f103959a640 10 monclient: dump: 2023-11-25T05:04:06.808 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:06.808 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:06.808 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:06.808 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:06.808 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:06.808 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:06.808 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:06.808 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:06.808 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:06.808 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:06.808 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.804+0000 7f1040d33640 10 monclient: shutdown 2023-11-25T05:04:06.808 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.804+0000 7f1040d33640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:06.809 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.804+0000 7f1040d33640 1 -- 172.21.15.7:0/2707658037 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f103c0a89e0 msgr2=0x7f103c13ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:06.809 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.804+0000 7f1040d33640 1 --2- 172.21.15.7:0/2707658037 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f103c0a89e0 0x7f103c13ef20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f102c009870 tx=0x7f102c02b8f0 comp rx=0 tx=0).stop 2023-11-25T05:04:06.809 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.804+0000 7f1040d33640 1 -- 172.21.15.7:0/2707658037 shutdown_connections 2023-11-25T05:04:06.809 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.804+0000 7f1040d33640 1 --2- 172.21.15.7:0/2707658037 >> v2:172.21.15.7:3300/0 conn(0x7f103c1428e0 0x7f103c14c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:06.809 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.804+0000 7f1040d33640 1 --2- 172.21.15.7:0/2707658037 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f103c0a89e0 0x7f103c13ef20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:06.809 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.804+0000 7f1040d33640 1 -- 172.21.15.7:0/2707658037 >> 172.21.15.7:0/2707658037 conn(0x7f103c0b7940 msgr2=0x7f103c0b7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:06.809 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.804+0000 7f1040d33640 1 -- 172.21.15.7:0/2707658037 shutdown_connections 2023-11-25T05:04:06.809 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.804+0000 7f1040d33640 1 -- 172.21.15.7:0/2707658037 wait complete. 2023-11-25T05:04:06.813 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.805+0000 7f1040d33640 10 monclient: build_initial_monmap 2023-11-25T05:04:06.813 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.805+0000 7f1040d33640 10 monclient: monmap: 2023-11-25T05:04:06.813 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-11-25T05:04:06.813 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:06.813 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:06.813 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2023-11-25T05:04:06.814 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:06.814 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:06.814 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:06.814 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:06.814 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:06.814 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:06.814 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.805+0000 7f1040d33640 1 librados: starting msgr at 2023-11-25T05:04:06.814 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.805+0000 7f1040d33640 1 librados: starting objecter 2023-11-25T05:04:06.814 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.805+0000 7f1040d33640 1 Processor -- start 2023-11-25T05:04:06.814 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.805+0000 7f1040d33640 1 -- start start 2023-11-25T05:04:06.814 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.805+0000 7f1040d33640 1 librados: setting wanted keys 2023-11-25T05:04:06.814 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.805+0000 7f1040d33640 1 librados: calling monclient init 2023-11-25T05:04:06.815 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.805+0000 7f1040d33640 10 monclient: init 2023-11-25T05:04:06.815 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:06.815 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:04:06.815 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f103c0a89e0 0x7f103c1374a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:06.815 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 10 monclient(hunting): picked mon.noname-a con 0x7f103c0a89e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:06.815 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f103c13f460 0x7f103c13fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:06.815 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 10 monclient(hunting): picked mon.noname-c con 0x7f103c13f460 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:06.815 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f103c1428e0 0x7f103c140230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:06.815 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 10 monclient(hunting): picked mon.noname-b con 0x7f103c1428e0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:06.815 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:06.815 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c145c30 con 0x7f103c0a89e0 2023-11-25T05:04:06.815 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:06.816 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c145370 con 0x7f103c1428e0 2023-11-25T05:04:06.816 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:06.816 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f103c140f90 con 0x7f103c13f460 2023-11-25T05:04:06.816 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:06.816 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1040d33640 10 monclient(hunting): authenticate will time out at 1373.049316s 2023-11-25T05:04:06.816 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.806+0000 7f1039d9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f103c13f460 0x7f103c13fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:06.816 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f103a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f103c0a89e0 0x7f103c1374a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:06.816 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f1039d9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f103c13f460 0x7f103c13fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:42156/0 (socket says 172.21.15.7:42156) 2023-11-25T05:04:06.816 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f1039d9b640 1 -- 172.21.15.7:0/2377857895 learned_addr learned my addr 172.21.15.7:0/2377857895 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:06.816 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f1039d9b640 10 monclient(hunting): get_auth_request con 0x7f103c13f460 auth_method 0 2023-11-25T05:04:06.816 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f1039d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:06.816 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f1039d9b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:06.816 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f1039d9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:06.817 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f103ad9d640 1 --2- 172.21.15.7:0/2377857895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f103c1428e0 0x7f103c140230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:06.817 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f103a59c640 10 monclient(hunting): get_auth_request con 0x7f103c0a89e0 auth_method 0 2023-11-25T05:04:06.817 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f103a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:06.817 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f103a59c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:06.817 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f103a59c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:06.817 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f1039d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:06.817 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f1039d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:06.817 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f1039d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:06.817 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f103ad9d640 10 monclient(hunting): get_auth_request con 0x7f103c1428e0 auth_method 0 2023-11-25T05:04:06.817 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f103ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:06.817 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f103ad9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:06.817 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.807+0000 7f103ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:06.818 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.808+0000 7f103a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:06.818 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.808+0000 7f103a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:06.818 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.808+0000 7f103a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:06.818 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.808+0000 7f103ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:06.818 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.808+0000 7f103ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:06.818 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.808+0000 7f103ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:06.818 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.811+0000 7f103a59c640 10 monclient(hunting): handle_auth_done global_id 4206 payload 824 2023-11-25T05:04:06.818 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.811+0000 7f103a59c640 1 -- 172.21.15.7:0/2377857895 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f103c13f460 msgr2=0x7f103c13fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:06.818 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.811+0000 7f103a59c640 1 --2- 172.21.15.7:0/2377857895 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f103c13f460 0x7f103c13fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:06.818 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.811+0000 7f103a59c640 1 -- 172.21.15.7:0/2377857895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f103c1428e0 msgr2=0x7f103c140230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:06.818 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.811+0000 7f103a59c640 1 --2- 172.21.15.7:0/2377857895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f103c1428e0 0x7f103c140230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:06.818 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.811+0000 7f103a59c640 10 monclient: _finish_hunting 0 2023-11-25T05:04:06.818 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.811+0000 7f103a59c640 1 monclient: found mon.noname-a 2023-11-25T05:04:06.819 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.811+0000 7f103a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:06.819 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.811+0000 7f103a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:06.819 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.811+0000 7f103a59c640 1 -- 172.21.15.7:0/2377857895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1034009a70 con 0x7f103c0a89e0 2023-11-25T05:04:06.821 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.813+0000 7f103a59c640 1 --2- 172.21.15.7:0/2377857895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f103c0a89e0 0x7f103c1374a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f102c033000 tx=0x7f102c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:06.821 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.813+0000 7f102affd640 10 client.?.objecter ms_handle_connect 0x7f103c0a89e0 2023-11-25T05:04:06.821 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.813+0000 7f102affd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:06.821 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f102affd640 1 -- 172.21.15.7:0/2377857895 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c03abe0 con 0x7f103c0a89e0 2023-11-25T05:04:06.821 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f102affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:06.821 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f102affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:06.821 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f102affd640 10 monclient: dump: 2023-11-25T05:04:06.821 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:06.821 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:06.821 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:06.821 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:06.822 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:06.822 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:06.822 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:06.822 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:06.822 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:06.822 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:06.822 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f102affd640 10 monclient: _finish_auth 0 2023-11-25T05:04:06.822 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f102affd640 10 monclient: _check_auth_tickets 2023-11-25T05:04:06.822 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f102affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:06.822 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f102affd640 1 -- 172.21.15.7:0/2377857895 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102c03ad60 con 0x7f103c0a89e0 2023-11-25T05:04:06.822 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f102affd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:06.822 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f102affd640 1 -- 172.21.15.7:0/2377857895 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c044630 con 0x7f103c0a89e0 2023-11-25T05:04:06.823 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f102affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:06.823 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f102affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:06.823 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f102affd640 10 monclient: dump: 2023-11-25T05:04:06.823 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:06.823 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:06.823 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:06.823 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:06.823 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:06.823 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:06.823 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:06.823 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:06.823 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:06.823 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:06.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f1040d33640 5 monclient: authenticate success, global_id 4206 2023-11-25T05:04:06.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f1038d99640 4 set_mon_vals no callback set 2023-11-25T05:04:06.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f1040d33640 10 monclient: _renew_subs 2023-11-25T05:04:06.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f1040d33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:06.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f1040d33640 1 -- 172.21.15.7:0/2377857895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102c0094d0 con 0x7f103c0a89e0 2023-11-25T05:04:06.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f1040d33640 10 client.4206.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:06.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f1040d33640 10 monclient: _renew_subs 2023-11-25T05:04:06.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f1040d33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:06.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f1040d33640 1 -- 172.21.15.7:0/2377857895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103c135810 con 0x7f103c0a89e0 2023-11-25T05:04:06.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.817+0000 7f1040d33640 1 librados: init done 2023-11-25T05:04:06.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.818+0000 7f1040d33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:06.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.818+0000 7f1040d33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:06.824 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.818+0000 7f1040d33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:06.825 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.818+0000 7f1040d33640 1 -- 172.21.15.7:0/2377857895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f103c135ac0 con 0x7f103c0a89e0 2023-11-25T05:04:06.840 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.838+0000 7f102affd640 1 -- 172.21.15.7:0/2377857895 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f102c02f050 con 0x7f103c0a89e0 2023-11-25T05:04:06.840 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.838+0000 7f102affd640 1 --2- 172.21.15.7:0/2377857895 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f101c06c790 0x7f101c06ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:06.841 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.839+0000 7f1039d9b640 1 --2- 172.21.15.7:0/2377857895 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f101c06c790 0x7f101c06ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:06.841 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.839+0000 7f1039d9b640 10 monclient: get_auth_request con 0x7f101c06c790 auth_method 0 2023-11-25T05:04:06.841 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.839+0000 7f1039d9b640 1 --2- 172.21.15.7:0/2377857895 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f101c06c790 0x7f101c06ec50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f103c141ec0 tx=0x7f1030009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:06.841 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.839+0000 7f102affd640 10 client.4206.objecter ms_handle_connect 0x7f101c06c790 2023-11-25T05:04:06.843 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.841+0000 7f102affd640 1 -- 172.21.15.7:0/2377857895 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f102c07ac20 con 0x7f103c0a89e0 2023-11-25T05:04:06.843 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.841+0000 7f102affd640 10 client.4206.objecter ms_dispatch 0x7f103c000b90 osd_map(10..10 src has 1..10) v4 2023-11-25T05:04:06.843 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.841+0000 7f102affd640 3 client.4206.objecter handle_osd_map got epochs [10,10] > 0 2023-11-25T05:04:06.843 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.841+0000 7f102affd640 3 client.4206.objecter handle_osd_map decoding full epoch 10 2023-11-25T05:04:06.843 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:06.841+0000 7f102affd640 20 client.4206.objecter dump_active .. 0 homeless 2023-11-25T05:04:06.953 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:25.029 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:06.957 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:25.033 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:07.193 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.192+0000 7f102affd640 1 -- 172.21.15.7:0/2377857895 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f102c076860 con 0x7f103c0a89e0 2023-11-25T05:04:07.193 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.192+0000 7f102affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:07.193 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.192+0000 7f102affd640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:07.435 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.433+0000 7f1040d33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:07.435 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.433+0000 7f1040d33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:07.435 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.433+0000 7f1040d33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:07.435 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.434+0000 7f1040d33640 1 -- 172.21.15.7:0/2377857895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f103c145a00 con 0x7f103c0a89e0 2023-11-25T05:04:07.460 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.458+0000 7f102affd640 1 -- 172.21.15.7:0/2377857895 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f102c079e80 con 0x7f103c0a89e0 2023-11-25T05:04:07.460 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.458+0000 7f102affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:07.460 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.458+0000 7f102affd640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:07.461 INFO:teuthology.misc.health.smithi007.stdout: 2023-11-25T05:04:07.461 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"59cf1eff-816d-4eda-b513-a2de7ea218f7","created":"2023-11-25T05:03:35.598582+0000","modified":"2023-11-25T05:03:41.565581+0000","last_up_change":"0.000000","last_in_change":"2023-11-25T05:03:41.565581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c87807e-7b82-4866-ba36-60b44a25f317","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0e252ab8-9524-4ccc-b08f-d95bbf05fbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccda96b8-aa9e-4c99-8d84-bba19ca0483b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e4b8700a-2e28-4e69-8577-465666b1b051","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"36c88a76-0467-4e37-85b4-dfd5953855a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ebf452a0-1b17-488a-bbea-2e953c932ac7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5aaf35c3-5c06-45e5-b788-8d3207c47cd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d50dd23a-f864-4593-ac00-4d8afed21dd0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-25T05:04:07.462 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 10 librados: watch_flush enter 2023-11-25T05:04:07.463 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 10 librados: watch_flush exit 2023-11-25T05:04:07.463 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 20 client.4206.objecter shutdown clearing up homeless session... 2023-11-25T05:04:07.463 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 10 client.4206.objecter successfully canceled tick 2023-11-25T05:04:07.463 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 1 -- 172.21.15.7:0/2377857895 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f101c06c790 msgr2=0x7f101c06ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:07.463 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 1 --2- 172.21.15.7:0/2377857895 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f101c06c790 0x7f101c06ec50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f103c141ec0 tx=0x7f1030009040 comp rx=0 tx=0).stop 2023-11-25T05:04:07.463 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 10 monclient: shutdown 2023-11-25T05:04:07.463 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:07.463 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 1 -- 172.21.15.7:0/2377857895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f103c0a89e0 msgr2=0x7f103c1374a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:07.463 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 1 --2- 172.21.15.7:0/2377857895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f103c0a89e0 0x7f103c1374a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f102c033000 tx=0x7f102c0046c0 comp rx=0 tx=0).stop 2023-11-25T05:04:07.463 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 1 -- 172.21.15.7:0/2377857895 shutdown_connections 2023-11-25T05:04:07.463 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 1 --2- 172.21.15.7:0/2377857895 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f101c06c790 0x7f101c06ec50 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:07.463 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 1 --2- 172.21.15.7:0/2377857895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f103c1428e0 0x7f103c140230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:07.463 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 1 --2- 172.21.15.7:0/2377857895 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f103c13f460 0x7f103c13fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:07.464 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 1 --2- 172.21.15.7:0/2377857895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f103c0a89e0 0x7f103c1374a0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f102c033000 tx=0x7f102c0046c0 comp rx=0 tx=0).stop 2023-11-25T05:04:07.464 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.461+0000 7f1040d33640 1 -- 172.21.15.7:0/2377857895 >> 172.21.15.7:0/2377857895 conn(0x7f103c0b7940 msgr2=0x7f103c0b7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:07.464 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.462+0000 7f1040d33640 1 -- 172.21.15.7:0/2377857895 shutdown_connections 2023-11-25T05:04:07.464 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.463+0000 7f1040d33640 1 -- 172.21.15.7:0/2377857895 wait complete. 2023-11-25T05:04:07.464 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:07.463+0000 7f1040d33640 1 librados: shutdown 2023-11-25T05:04:07.478 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-25T05:04:08.866 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:26.916 86838== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:08.953 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:26.985 86840== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:09.564 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:27.615 86837== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:09.669 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:27.713 86839== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:10.113 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:28.198 87159== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:10.938 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:29.026 87155== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:11.055 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:29.140 87158== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:11.774 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:29.850 87160== Warning: unimplemented fcntl command: 1038 2023-11-25T05:04:14.285 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-25T05:04:14.376 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.375+0000 7fc68fddc640 10 monclient: get_monmap_and_config 2023-11-25T05:04:14.377 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.375+0000 7fc68fddc640 10 monclient: build_initial_monmap 2023-11-25T05:04:14.377 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.375+0000 7fc68fddc640 10 monclient: monmap: 2023-11-25T05:04:14.377 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-11-25T05:04:14.377 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:14.377 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:04:14.376088+0000 2023-11-25T05:04:14.378 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:04:14.376088+0000 2023-11-25T05:04:14.378 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:14.378 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:14.378 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:14.379 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:14.379 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:14.379 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:14.379 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:14.379 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:14.383 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.375+0000 7fc68fddc640 1 Processor -- start 2023-11-25T05:04:14.383 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.376+0000 7fc68fddc640 1 -- start start 2023-11-25T05:04:14.383 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.376+0000 7fc68fddc640 10 monclient: init 2023-11-25T05:04:14.383 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.376+0000 7fc68fddc640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:14.383 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.376+0000 7fc68fddc640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-11-25T05:04:14.383 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.376+0000 7fc68fddc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6880a89e0 0x7fc68813ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:14.383 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.376+0000 7fc68fddc640 10 monclient(hunting): picked mon.noname-b con 0x7fc6880a89e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:14.383 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.376+0000 7fc68fddc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc68813f460 0x7fc6881421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:14.384 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.376+0000 7fc68fddc640 10 monclient(hunting): picked mon.noname-a con 0x7fc68813f460 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:14.384 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.376+0000 7fc68fddc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc688142710 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:14.384 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.376+0000 7fc68fddc640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6880aa770 con 0x7fc688142710 2023-11-25T05:04:14.384 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.377+0000 7fc68fddc640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:14.384 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.377+0000 7fc68fddc640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc68813f860 con 0x7fc68813f460 2023-11-25T05:04:14.384 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.377+0000 7fc68fddc640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:14.384 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.377+0000 7fc68fddc640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6880aa9c0 con 0x7fc6880a89e0 2023-11-25T05:04:14.384 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.377+0000 7fc68fddc640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:14.384 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.377+0000 7fc68fddc640 10 monclient(hunting): authenticate will time out at 1380.619845s 2023-11-25T05:04:14.384 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.377+0000 7fc68db78640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6880a89e0 0x7fc68813ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:14.384 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.377+0000 7fc68e379640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc688142710 0x7fc688145470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58608/0 (socket says 172.21.15.7:58608) 2023-11-25T05:04:14.385 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.377+0000 7fc68e379640 1 -- 172.21.15.7:0/2624498593 learned_addr learned my addr 172.21.15.7:0/2624498593 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:14.385 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.377+0000 7fc68d377640 1 --2- 172.21.15.7:0/2624498593 >> v2:172.21.15.7:3300/0 conn(0x7fc68813f460 0x7fc6881421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:14.385 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.377+0000 7fc68db78640 10 monclient(hunting): get_auth_request con 0x7fc6880a89e0 auth_method 0 2023-11-25T05:04:14.385 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.377+0000 7fc68db78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:14.385 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.377+0000 7fc68db78640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:14.385 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.377+0000 7fc68db78640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:14.385 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.378+0000 7fc68db78640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:14.385 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.378+0000 7fc68db78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:14.385 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.378+0000 7fc68db78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:14.385 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.378+0000 7fc68d377640 10 monclient(hunting): get_auth_request con 0x7fc68813f460 auth_method 0 2023-11-25T05:04:14.385 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.378+0000 7fc68d377640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:14.385 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.378+0000 7fc68d377640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:14.386 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.378+0000 7fc68d377640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:14.386 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.380+0000 7fc68d377640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:14.386 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.380+0000 7fc68d377640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:14.386 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.380+0000 7fc68d377640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:14.386 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.380+0000 7fc68db78640 10 monclient(hunting): handle_auth_done global_id 4187 payload 274 2023-11-25T05:04:14.386 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.380+0000 7fc68db78640 1 -- 172.21.15.7:0/2624498593 >> v2:172.21.15.7:3300/0 conn(0x7fc68813f460 msgr2=0x7fc6881421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:14.386 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.380+0000 7fc68db78640 1 --2- 172.21.15.7:0/2624498593 >> v2:172.21.15.7:3300/0 conn(0x7fc68813f460 0x7fc6881421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:14.386 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.380+0000 7fc68db78640 1 -- 172.21.15.7:0/2624498593 >> v1:172.21.15.7:6789/0 conn(0x7fc688142710 legacy=0x7fc688145470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:14.386 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.380+0000 7fc68db78640 10 monclient: _finish_hunting 0 2023-11-25T05:04:14.386 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.380+0000 7fc68db78640 1 monclient: found mon.noname-b 2023-11-25T05:04:14.386 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.380+0000 7fc68db78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:14.387 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.380+0000 7fc68db78640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:04:14.387 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.380+0000 7fc68db78640 1 -- 172.21.15.7:0/2624498593 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6880aaca0 con 0x7fc6880a89e0 2023-11-25T05:04:14.387 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.383+0000 7fc68db78640 1 --2- 172.21.15.7:0/2624498593 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6880a89e0 0x7fc68813ef20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc6840097a0 tx=0x7fc68402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=848d4d561f6db05e server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:14.396 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68cb76640 1 -- 172.21.15.7:0/2624498593 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc684031780 con 0x7fc6880a89e0 2023-11-25T05:04:14.396 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68cb76640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:14.396 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68cb76640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:14.396 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68cb76640 10 monclient: dump: 2023-11-25T05:04:14.396 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:14.396 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:14.397 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:14.397 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:14.397 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:14.397 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:14.397 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:14.397 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:14.397 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:14.397 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:14.397 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68cb76640 10 monclient: _finish_auth 0 2023-11-25T05:04:14.397 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68cb76640 10 monclient: _check_auth_tickets 2023-11-25T05:04:14.397 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68cb76640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:14.397 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68cb76640 1 -- 172.21.15.7:0/2624498593 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6840318e0 con 0x7fc6880a89e0 2023-11-25T05:04:14.398 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68cb76640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:14.398 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68cb76640 1 -- 172.21.15.7:0/2624498593 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc684031c70 con 0x7fc6880a89e0 2023-11-25T05:04:14.398 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68cb76640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:14.398 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68cb76640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:14.398 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68cb76640 10 monclient: dump: 2023-11-25T05:04:14.398 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:14.398 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:14.398 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:14.398 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:14.398 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:14.398 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:14.398 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:14.399 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:14.399 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:14.399 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:14.399 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68fddc640 5 monclient: authenticate success, global_id 4187 2023-11-25T05:04:14.399 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68fddc640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:14.399 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.388+0000 7fc68fddc640 4 set_mon_vals no callback set 2023-11-25T05:04:14.399 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.389+0000 7fc68fddc640 10 monclient: shutdown 2023-11-25T05:04:14.399 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.389+0000 7fc68fddc640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:14.399 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.389+0000 7fc68fddc640 1 -- 172.21.15.7:0/2624498593 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6880a89e0 msgr2=0x7fc68813ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:14.399 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.389+0000 7fc68fddc640 1 --2- 172.21.15.7:0/2624498593 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6880a89e0 0x7fc68813ef20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc6840097a0 tx=0x7fc68402b820 comp rx=0 tx=0).stop 2023-11-25T05:04:14.399 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.389+0000 7fc68fddc640 1 -- 172.21.15.7:0/2624498593 shutdown_connections 2023-11-25T05:04:14.400 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.389+0000 7fc68fddc640 1 --2- 172.21.15.7:0/2624498593 >> v2:172.21.15.7:3300/0 conn(0x7fc68813f460 0x7fc6881421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:14.400 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.389+0000 7fc68fddc640 1 --2- 172.21.15.7:0/2624498593 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6880a89e0 0x7fc68813ef20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:14.400 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.389+0000 7fc68fddc640 1 -- 172.21.15.7:0/2624498593 >> 172.21.15.7:0/2624498593 conn(0x7fc6880b7940 msgr2=0x7fc6880b7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:14.400 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.390+0000 7fc68fddc640 1 -- 172.21.15.7:0/2624498593 shutdown_connections 2023-11-25T05:04:14.400 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.390+0000 7fc68fddc640 1 -- 172.21.15.7:0/2624498593 wait complete. 2023-11-25T05:04:14.400 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.390+0000 7fc68fddc640 10 monclient: build_initial_monmap 2023-11-25T05:04:14.400 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.390+0000 7fc68fddc640 10 monclient: monmap: 2023-11-25T05:04:14.400 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-11-25T05:04:14.400 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:14.400 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:14.400 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2023-11-25T05:04:14.400 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:14.401 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:14.401 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:14.401 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:14.401 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:14.401 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:14.401 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.391+0000 7fc68fddc640 1 librados: starting msgr at 2023-11-25T05:04:14.401 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.391+0000 7fc68fddc640 1 librados: starting objecter 2023-11-25T05:04:14.402 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.391+0000 7fc68fddc640 1 Processor -- start 2023-11-25T05:04:14.402 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.391+0000 7fc68fddc640 1 -- start start 2023-11-25T05:04:14.402 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.391+0000 7fc68fddc640 1 librados: setting wanted keys 2023-11-25T05:04:14.402 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.391+0000 7fc68fddc640 1 librados: calling monclient init 2023-11-25T05:04:14.402 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.391+0000 7fc68fddc640 10 monclient: init 2023-11-25T05:04:14.402 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.391+0000 7fc68fddc640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:14.403 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.391+0000 7fc68fddc640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:04:14.403 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.391+0000 7fc68fddc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6880a89e0 0x7fc68813e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:14.403 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.391+0000 7fc68fddc640 10 monclient(hunting): picked mon.noname-c con 0x7fc6880a89e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:14.404 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.392+0000 7fc68fddc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc68813f460 0x7fc688141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:14.404 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.392+0000 7fc68fddc640 10 monclient(hunting): picked mon.noname-b con 0x7fc68813f460 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:14.404 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.392+0000 7fc68fddc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc688142710 0x7fc688145930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:14.404 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.392+0000 7fc68fddc640 10 monclient(hunting): picked mon.noname-a con 0x7fc688142710 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:14.404 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.392+0000 7fc68fddc640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:14.404 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.392+0000 7fc68fddc640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc688145450 con 0x7fc688142710 2023-11-25T05:04:14.404 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.392+0000 7fc68fddc640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:14.404 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.392+0000 7fc68fddc640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6881433a0 con 0x7fc68813f460 2023-11-25T05:04:14.404 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.392+0000 7fc68fddc640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:14.404 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.392+0000 7fc68fddc640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc688143870 con 0x7fc6880a89e0 2023-11-25T05:04:14.404 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.392+0000 7fc68fddc640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:14.404 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.392+0000 7fc68fddc640 10 monclient(hunting): authenticate will time out at 1380.635643s 2023-11-25T05:04:14.405 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.392+0000 7fc68d377640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc68813f460 0x7fc688141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:14.405 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68e379640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc688142710 0x7fc688145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:14.405 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68db78640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6880a89e0 0x7fc68813e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:14.405 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68db78640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6880a89e0 0x7fc68813e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:42184/0 (socket says 172.21.15.7:42184) 2023-11-25T05:04:14.405 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68db78640 1 -- 172.21.15.7:0/1449420736 learned_addr learned my addr 172.21.15.7:0/1449420736 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:14.405 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68db78640 10 monclient(hunting): get_auth_request con 0x7fc6880a89e0 auth_method 0 2023-11-25T05:04:14.405 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68db78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:14.405 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68db78640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:14.405 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68db78640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:14.406 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68e379640 10 monclient(hunting): get_auth_request con 0x7fc688142710 auth_method 0 2023-11-25T05:04:14.406 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68e379640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:14.406 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68e379640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:14.406 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68e379640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:14.406 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68d377640 10 monclient(hunting): get_auth_request con 0x7fc68813f460 auth_method 0 2023-11-25T05:04:14.406 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68d377640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:14.406 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68d377640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:14.406 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68d377640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:14.406 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.393+0000 7fc68db78640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:14.406 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.394+0000 7fc68db78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:14.406 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.394+0000 7fc68db78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:14.406 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.394+0000 7fc68e379640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:14.407 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.394+0000 7fc68e379640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:14.407 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.394+0000 7fc68e379640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:14.407 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.394+0000 7fc68d377640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:14.407 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.394+0000 7fc68d377640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:14.407 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.394+0000 7fc68d377640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:14.411 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.397+0000 7fc68d377640 10 monclient(hunting): handle_auth_done global_id 4216 payload 824 2023-11-25T05:04:14.411 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.397+0000 7fc68d377640 1 -- 172.21.15.7:0/1449420736 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6880a89e0 msgr2=0x7fc68813e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:14.411 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.397+0000 7fc68d377640 1 --2- 172.21.15.7:0/1449420736 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6880a89e0 0x7fc68813e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:14.411 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.397+0000 7fc68d377640 1 -- 172.21.15.7:0/1449420736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc688142710 msgr2=0x7fc688145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:14.411 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.397+0000 7fc68d377640 1 --2- 172.21.15.7:0/1449420736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc688142710 0x7fc688145930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:14.412 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.397+0000 7fc68d377640 10 monclient: _finish_hunting 0 2023-11-25T05:04:14.412 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.397+0000 7fc68d377640 1 monclient: found mon.noname-b 2023-11-25T05:04:14.412 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.397+0000 7fc68d377640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:14.412 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.397+0000 7fc68d377640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:14.412 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.397+0000 7fc68d377640 1 -- 172.21.15.7:0/1449420736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc684009400 con 0x7fc68813f460 2023-11-25T05:04:14.412 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.401+0000 7fc68d377640 1 --2- 172.21.15.7:0/1449420736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc68813f460 0x7fc688141310 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc67800e850 tx=0x7fc67800ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:14.412 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.401+0000 7fc67e7fc640 10 client.?.objecter ms_handle_connect 0x7fc68813f460 2023-11-25T05:04:14.412 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.401+0000 7fc67e7fc640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:14.412 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc67e7fc640 1 -- 172.21.15.7:0/1449420736 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6780095b0 con 0x7fc68813f460 2023-11-25T05:04:14.412 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc67e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:14.412 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc67e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:14.412 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc67e7fc640 10 monclient: dump: 2023-11-25T05:04:14.413 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:14.413 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:14.413 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:14.413 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:14.413 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:14.413 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:14.413 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:14.413 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:14.413 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:14.413 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:14.413 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc67e7fc640 10 monclient: _finish_auth 0 2023-11-25T05:04:14.413 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc67e7fc640 10 monclient: _check_auth_tickets 2023-11-25T05:04:14.413 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc67e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:14.414 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc67e7fc640 1 -- 172.21.15.7:0/1449420736 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67800c940 con 0x7fc68813f460 2023-11-25T05:04:14.414 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc68fddc640 5 monclient: authenticate success, global_id 4216 2023-11-25T05:04:14.414 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc68fddc640 10 monclient: _renew_subs 2023-11-25T05:04:14.414 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc68fddc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:14.414 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc68fddc640 1 -- 172.21.15.7:0/1449420736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc688143b50 con 0x7fc68813f460 2023-11-25T05:04:14.414 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc67e7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:14.414 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc67ffff640 4 set_mon_vals no callback set 2023-11-25T05:04:14.414 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc68fddc640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:14.414 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc67e7fc640 1 -- 172.21.15.7:0/1449420736 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67800ccd0 con 0x7fc68813f460 2023-11-25T05:04:14.414 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc67e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:14.414 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc67e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:14.414 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.406+0000 7fc67e7fc640 10 monclient: dump: 2023-11-25T05:04:14.415 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:14.415 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:14.415 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:14.415 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:14.415 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:14.415 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:14.415 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:14.415 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:14.415 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:14.415 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:14.415 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.407+0000 7fc68fddc640 10 monclient: _renew_subs 2023-11-25T05:04:14.416 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.407+0000 7fc68fddc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:14.416 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.407+0000 7fc68fddc640 1 -- 172.21.15.7:0/1449420736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc688146050 con 0x7fc68813f460 2023-11-25T05:04:14.416 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.407+0000 7fc68fddc640 1 librados: init done 2023-11-25T05:04:14.416 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.408+0000 7fc68fddc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:14.416 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.408+0000 7fc68fddc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:14.416 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.408+0000 7fc68fddc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:14.416 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.408+0000 7fc68fddc640 1 -- 172.21.15.7:0/1449420736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6881462a0 con 0x7fc68813f460 2023-11-25T05:04:14.441 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.439+0000 7fc67e7fc640 1 -- 172.21.15.7:0/1449420736 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7fc678013470 con 0x7fc68813f460 2023-11-25T05:04:14.441 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.440+0000 7fc67e7fc640 1 --2- 172.21.15.7:0/1449420736 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc66c06c740 0x7fc66c06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:14.442 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.440+0000 7fc68db78640 1 --2- 172.21.15.7:0/1449420736 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc66c06c740 0x7fc66c06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:14.442 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.440+0000 7fc68db78640 10 monclient: get_auth_request con 0x7fc66c06c740 auth_method 0 2023-11-25T05:04:14.442 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.440+0000 7fc67e7fc640 10 client.4216.objecter ms_handle_connect 0x7fc66c06c740 2023-11-25T05:04:14.442 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.440+0000 7fc68db78640 1 --2- 172.21.15.7:0/1449420736 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc66c06c740 0x7fc66c06ec00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc684000c00 tx=0x7fc68403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:14.884 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.881+0000 7fc67e7fc640 1 -- 172.21.15.7:0/1449420736 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc67800ce30 con 0x7fc68813f460 2023-11-25T05:04:14.884 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.881+0000 7fc67e7fc640 10 client.4216.objecter ms_dispatch 0x7fc688000b90 osd_map(10..10 src has 1..10) v4 2023-11-25T05:04:14.884 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.881+0000 7fc67e7fc640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2023-11-25T05:04:14.884 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.881+0000 7fc67e7fc640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2023-11-25T05:04:14.884 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.881+0000 7fc67e7fc640 20 client.4216.objecter dump_active .. 0 homeless 2023-11-25T05:04:14.885 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.884+0000 7fc67e7fc640 1 -- 172.21.15.7:0/1449420736 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc678054cf0 con 0x7fc68813f460 2023-11-25T05:04:14.885 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.884+0000 7fc67e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:14.885 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:14.884+0000 7fc67e7fc640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:15.226 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.224+0000 7fc68fddc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:15.226 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.224+0000 7fc68fddc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:15.226 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.224+0000 7fc68fddc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:15.226 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.224+0000 7fc68fddc640 1 -- 172.21.15.7:0/1449420736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6881462a0 con 0x7fc68813f460 2023-11-25T05:04:15.250 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.249+0000 7fc67e7fc640 1 -- 172.21.15.7:0/1449420736 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fc6881462a0 con 0x7fc68813f460 2023-11-25T05:04:15.250 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.249+0000 7fc67e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:15.251 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.249+0000 7fc67e7fc640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:15.251 INFO:teuthology.misc.health.smithi007.stdout: 2023-11-25T05:04:15.251 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"59cf1eff-816d-4eda-b513-a2de7ea218f7","created":"2023-11-25T05:03:35.598582+0000","modified":"2023-11-25T05:03:41.565581+0000","last_up_change":"0.000000","last_in_change":"2023-11-25T05:03:41.565581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c87807e-7b82-4866-ba36-60b44a25f317","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0e252ab8-9524-4ccc-b08f-d95bbf05fbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccda96b8-aa9e-4c99-8d84-bba19ca0483b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e4b8700a-2e28-4e69-8577-465666b1b051","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"36c88a76-0467-4e37-85b4-dfd5953855a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ebf452a0-1b17-488a-bbea-2e953c932ac7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5aaf35c3-5c06-45e5-b788-8d3207c47cd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d50dd23a-f864-4593-ac00-4d8afed21dd0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-25T05:04:15.253 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.251+0000 7fc68fddc640 10 librados: watch_flush enter 2023-11-25T05:04:15.253 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.251+0000 7fc68fddc640 10 librados: watch_flush exit 2023-11-25T05:04:15.254 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.251+0000 7fc68fddc640 20 client.4216.objecter shutdown clearing up homeless session... 2023-11-25T05:04:15.254 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.251+0000 7fc68fddc640 10 client.4216.objecter successfully canceled tick 2023-11-25T05:04:15.254 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.251+0000 7fc68fddc640 1 -- 172.21.15.7:0/1449420736 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc66c06c740 msgr2=0x7fc66c06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:15.254 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.251+0000 7fc68fddc640 1 --2- 172.21.15.7:0/1449420736 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc66c06c740 0x7fc66c06ec00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc684000c00 tx=0x7fc68403d040 comp rx=0 tx=0).stop 2023-11-25T05:04:15.254 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.251+0000 7fc68fddc640 10 monclient: shutdown 2023-11-25T05:04:15.254 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.251+0000 7fc68fddc640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:15.254 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.251+0000 7fc68fddc640 1 -- 172.21.15.7:0/1449420736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc68813f460 msgr2=0x7fc688141310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:15.255 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.251+0000 7fc68fddc640 1 --2- 172.21.15.7:0/1449420736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc68813f460 0x7fc688141310 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc67800e850 tx=0x7fc67800ed10 comp rx=0 tx=0).stop 2023-11-25T05:04:15.255 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.252+0000 7fc68fddc640 1 -- 172.21.15.7:0/1449420736 shutdown_connections 2023-11-25T05:04:15.255 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.252+0000 7fc68fddc640 1 --2- 172.21.15.7:0/1449420736 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc66c06c740 0x7fc66c06ec00 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:15.255 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.252+0000 7fc68fddc640 1 --2- 172.21.15.7:0/1449420736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc688142710 0x7fc688145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:15.255 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.252+0000 7fc68fddc640 1 --2- 172.21.15.7:0/1449420736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc68813f460 0x7fc688141310 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:15.255 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.252+0000 7fc68fddc640 1 --2- 172.21.15.7:0/1449420736 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6880a89e0 0x7fc68813e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:15.255 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.252+0000 7fc68fddc640 1 -- 172.21.15.7:0/1449420736 >> 172.21.15.7:0/1449420736 conn(0x7fc6880b7940 msgr2=0x7fc6880b18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:15.256 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.252+0000 7fc68fddc640 1 -- 172.21.15.7:0/1449420736 shutdown_connections 2023-11-25T05:04:15.256 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.252+0000 7fc68fddc640 1 -- 172.21.15.7:0/1449420736 wait complete. 2023-11-25T05:04:15.256 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:15.252+0000 7fc68fddc640 1 librados: shutdown 2023-11-25T05:04:15.264 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-25T05:04:17.136 INFO:tasks.ceph.osd.5.smithi090.stderr:2023-11-25T05:04:17.132+0000 5dc2340 -1 osd.5 0 log_to_monitors true 2023-11-25T05:04:17.605 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-25T05:04:17.599+0000 5dc2340 -1 osd.7 0 log_to_monitors true 2023-11-25T05:04:17.747 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:04:17.741+0000 5dc2340 -1 osd.4 0 log_to_monitors true 2023-11-25T05:04:18.338 INFO:tasks.ceph.osd.6.smithi090.stderr:2023-11-25T05:04:18.333+0000 5dc2340 -1 osd.6 0 log_to_monitors true 2023-11-25T05:04:18.750 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:04:18.745+0000 5dc2340 -1 osd.2 0 log_to_monitors true 2023-11-25T05:04:19.656 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-25T05:04:19.651+0000 5dc2340 -1 osd.1 0 log_to_monitors true 2023-11-25T05:04:19.832 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-25T05:04:19.825+0000 5dc2340 -1 osd.0 0 log_to_monitors true 2023-11-25T05:04:20.330 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-11-25T05:04:20.323+0000 5dc2340 -1 osd.3 0 log_to_monitors true 2023-11-25T05:04:20.529 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-25T05:04:20.515+0000 c493640 -1 osd.7 0 waiting for initial osdmap 2023-11-25T05:04:20.556 INFO:tasks.ceph.osd.5.smithi090.stderr:2023-11-25T05:04:20.547+0000 c493640 -1 osd.5 0 waiting for initial osdmap 2023-11-25T05:04:20.923 INFO:tasks.ceph.osd.5.smithi090.stderr:2023-11-25T05:04:20.909+0000 12256640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-25T05:04:20.938 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-25T05:04:20.922+0000 11a55640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-25T05:04:21.746 INFO:tasks.ceph.osd.6.smithi090.stderr:2023-11-25T05:04:21.740+0000 c493640 -1 osd.6 0 waiting for initial osdmap 2023-11-25T05:04:21.759 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:04:21.752+0000 c493640 -1 osd.4 0 waiting for initial osdmap 2023-11-25T05:04:22.070 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-25T05:04:22.141 INFO:tasks.ceph.osd.6.smithi090.stderr:2023-11-25T05:04:22.126+0000 11a55640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-25T05:04:22.167 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.164+0000 7fec31d68640 10 monclient: get_monmap_and_config 2023-11-25T05:04:22.167 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.164+0000 7fec31d68640 10 monclient: build_initial_monmap 2023-11-25T05:04:22.167 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.164+0000 7fec31d68640 10 monclient: monmap: 2023-11-25T05:04:22.167 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-11-25T05:04:22.168 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:22.168 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:04:22.166898+0000 2023-11-25T05:04:22.168 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:04:22.166898+0000 2023-11-25T05:04:22.168 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:22.168 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:22.168 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:22.168 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:22.168 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:22.168 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:22.168 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:22.168 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:22.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.165+0000 7fec31d68640 1 Processor -- start 2023-11-25T05:04:22.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.165+0000 7fec31d68640 1 -- start start 2023-11-25T05:04:22.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.165+0000 7fec31d68640 10 monclient: init 2023-11-25T05:04:22.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.166+0000 7fec31d68640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:22.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.166+0000 7fec31d68640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-11-25T05:04:22.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.166+0000 7fec31d68640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fec2c0a8a00 0x7fec2c0a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:22.170 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.166+0000 7fec31d68640 10 monclient(hunting): picked mon.noname-b con 0x7fec2c0a8a00 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:22.171 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.166+0000 7fec31d68640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fec2c145a40 0x7fec2c145e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:22.171 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.166+0000 7fec31d68640 10 monclient(hunting): picked mon.noname-c con 0x7fec2c145a40 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:22.171 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.166+0000 7fec31d68640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec2c141f20 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:22.171 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.166+0000 7fec31d68640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c1463c0 con 0x7fec2c141f20 2023-11-25T05:04:22.171 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.166+0000 7fec31d68640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:22.171 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.166+0000 7fec31d68640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec2c1455b0 con 0x7fec2c145a40 2023-11-25T05:04:22.171 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.166+0000 7fec31d68640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:22.171 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.166+0000 7fec31d68640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c0aa970 con 0x7fec2c0a8a00 2023-11-25T05:04:22.171 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.166+0000 7fec31d68640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:22.171 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.166+0000 7fec31d68640 10 monclient(hunting): authenticate will time out at 1388.410254s 2023-11-25T05:04:22.171 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.167+0000 7fec2affd640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fec2c145a40 0x7fec2c145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:22.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.167+0000 7fec2bfff640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fec2c141f20 0x7fec2c1423b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:48068/0 (socket says 172.21.15.7:48068) 2023-11-25T05:04:22.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.167+0000 7fec2bfff640 1 -- 172.21.15.7:0/778015514 learned_addr learned my addr 172.21.15.7:0/778015514 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:22.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.168+0000 7fec2affd640 10 monclient(hunting): get_auth_request con 0x7fec2c145a40 auth_method 0 2023-11-25T05:04:22.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.168+0000 7fec2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:22.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.168+0000 7fec2affd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:22.172 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.168+0000 7fec2affd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:22.173 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.168+0000 7fec2affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:22.173 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.168+0000 7fec2affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:22.173 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.168+0000 7fec2affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:22.173 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.169+0000 7fec2b7fe640 1 --2- 172.21.15.7:0/778015514 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fec2c0a8a00 0x7fec2c0a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:22.173 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.170+0000 7fec2b7fe640 10 monclient(hunting): get_auth_request con 0x7fec2c0a8a00 auth_method 0 2023-11-25T05:04:22.173 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.170+0000 7fec2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:22.173 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.170+0000 7fec2b7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:22.173 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.170+0000 7fec2b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:22.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.171+0000 7fec2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:22.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.171+0000 7fec2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:22.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.171+0000 7fec2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:22.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.171+0000 7fec2affd640 10 monclient(hunting): handle_auth_done global_id 4243 payload 274 2023-11-25T05:04:22.174 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.171+0000 7fec2affd640 1 -- 172.21.15.7:0/778015514 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fec2c0a8a00 msgr2=0x7fec2c0a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:22.175 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.171+0000 7fec2affd640 1 --2- 172.21.15.7:0/778015514 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fec2c0a8a00 0x7fec2c0a8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:22.175 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.171+0000 7fec2affd640 1 -- 172.21.15.7:0/778015514 >> v1:172.21.15.90:6789/0 conn(0x7fec2c141f20 legacy=0x7fec2c1423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:22.175 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.171+0000 7fec2affd640 10 monclient: _finish_hunting 0 2023-11-25T05:04:22.175 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.171+0000 7fec2affd640 1 monclient: found mon.noname-c 2023-11-25T05:04:22.175 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.171+0000 7fec2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:22.175 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.171+0000 7fec2affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:22.175 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.171+0000 7fec2affd640 1 -- 172.21.15.7:0/778015514 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c0aac50 con 0x7fec2c145a40 2023-11-25T05:04:22.179 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.177+0000 7fec2affd640 1 --2- 172.21.15.7:0/778015514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec2c145a40 0x7fec2c145e80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fec200097a0 tx=0x7fec2002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6f776d165594c1c server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:22.191 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec2a7fc640 1 -- 172.21.15.7:0/778015514 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec20031830 con 0x7fec2c145a40 2023-11-25T05:04:22.191 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:22.191 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec2a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:22.192 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec2a7fc640 10 monclient: dump: 2023-11-25T05:04:22.192 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:22.192 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:22.192 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:22.192 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:22.192 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:22.192 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:22.193 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:22.193 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:22.193 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:22.193 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:22.193 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec2a7fc640 10 monclient: _finish_auth 0 2023-11-25T05:04:22.193 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec2a7fc640 10 monclient: _check_auth_tickets 2023-11-25T05:04:22.193 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:22.193 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec2a7fc640 1 -- 172.21.15.7:0/778015514 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec20031990 con 0x7fec2c145a40 2023-11-25T05:04:22.193 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec2a7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:22.193 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec2a7fc640 1 -- 172.21.15.7:0/778015514 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec20031d20 con 0x7fec2c145a40 2023-11-25T05:04:22.193 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:22.194 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec2a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:22.194 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec2a7fc640 10 monclient: dump: 2023-11-25T05:04:22.194 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:22.194 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:22.194 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:22.194 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:22.194 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:22.194 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:22.194 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:22.194 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:22.194 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:22.195 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:22.195 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec31d68640 5 monclient: authenticate success, global_id 4243 2023-11-25T05:04:22.195 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec31d68640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:22.195 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.187+0000 7fec31d68640 4 set_mon_vals no callback set 2023-11-25T05:04:22.195 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.188+0000 7fec31d68640 10 monclient: shutdown 2023-11-25T05:04:22.195 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.188+0000 7fec31d68640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:22.195 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.188+0000 7fec31d68640 1 -- 172.21.15.7:0/778015514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec2c145a40 msgr2=0x7fec2c145e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:22.195 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.188+0000 7fec31d68640 1 --2- 172.21.15.7:0/778015514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec2c145a40 0x7fec2c145e80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fec200097a0 tx=0x7fec2002b8d0 comp rx=0 tx=0).stop 2023-11-25T05:04:22.195 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.188+0000 7fec31d68640 1 -- 172.21.15.7:0/778015514 shutdown_connections 2023-11-25T05:04:22.195 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.188+0000 7fec31d68640 1 --2- 172.21.15.7:0/778015514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec2c145a40 0x7fec2c145e80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:22.196 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.188+0000 7fec31d68640 1 --2- 172.21.15.7:0/778015514 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fec2c0a8a00 0x7fec2c0a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:22.196 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.188+0000 7fec31d68640 1 -- 172.21.15.7:0/778015514 >> 172.21.15.7:0/778015514 conn(0x7fec2c066e80 msgr2=0x7fec2c0b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:22.196 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.188+0000 7fec31d68640 1 -- 172.21.15.7:0/778015514 shutdown_connections 2023-11-25T05:04:22.196 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.188+0000 7fec31d68640 1 -- 172.21.15.7:0/778015514 wait complete. 2023-11-25T05:04:22.196 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.188+0000 7fec31d68640 10 monclient: build_initial_monmap 2023-11-25T05:04:22.196 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.188+0000 7fec31d68640 10 monclient: monmap: 2023-11-25T05:04:22.196 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-11-25T05:04:22.196 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:22.200 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:22.200 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2023-11-25T05:04:22.200 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:22.200 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:22.200 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:22.200 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:22.200 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:22.200 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:22.201 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.189+0000 7fec31d68640 1 librados: starting msgr at 2023-11-25T05:04:22.201 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.189+0000 7fec31d68640 1 librados: starting objecter 2023-11-25T05:04:22.201 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.189+0000 7fec31d68640 1 Processor -- start 2023-11-25T05:04:22.201 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:04:22.178+0000 11a55640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-25T05:04:22.206 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.189+0000 7fec31d68640 1 -- start start 2023-11-25T05:04:22.207 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.189+0000 7fec31d68640 1 librados: setting wanted keys 2023-11-25T05:04:22.207 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.189+0000 7fec31d68640 1 librados: calling monclient init 2023-11-25T05:04:22.207 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.189+0000 7fec31d68640 10 monclient: init 2023-11-25T05:04:22.207 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:22.207 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-25T05:04:22.207 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec2c0a8a00 0x7fec2c0ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:22.207 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 10 monclient(hunting): picked mon.noname-b con 0x7fec2c0a8a00 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:22.207 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec2c141f20 0x7fec2c0ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:22.207 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 10 monclient(hunting): picked mon.noname-a con 0x7fec2c141f20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:22.207 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fec2c145a40 0x7fec2c1d6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:22.207 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 10 monclient(hunting): picked mon.noname-c con 0x7fec2c145a40 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:22.207 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:22.208 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c0aba00 con 0x7fec2c141f20 2023-11-25T05:04:22.208 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:22.208 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c0b7070 con 0x7fec2c0a8a00 2023-11-25T05:04:22.208 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:22.208 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c0ab690 con 0x7fec2c145a40 2023-11-25T05:04:22.208 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:22.208 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec31d68640 10 monclient(hunting): authenticate will time out at 1388.433952s 2023-11-25T05:04:22.208 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec2bfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fec2c145a40 0x7fec2c1d6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:22.208 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec2b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec2c0a8a00 0x7fec2c0ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:22.208 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec2bfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fec2c145a40 0x7fec2c1d6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:47788/0 (socket says 172.21.15.7:47788) 2023-11-25T05:04:22.208 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec2bfff640 1 -- 172.21.15.7:0/3950225591 learned_addr learned my addr 172.21.15.7:0/3950225591 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:22.208 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec2bfff640 10 monclient(hunting): get_auth_request con 0x7fec2c145a40 auth_method 0 2023-11-25T05:04:22.209 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:22.209 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec2bfff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:22.209 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.190+0000 7fec2bfff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:22.209 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.191+0000 7fec2b7fe640 10 monclient(hunting): get_auth_request con 0x7fec2c0a8a00 auth_method 0 2023-11-25T05:04:22.209 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.191+0000 7fec2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:22.209 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.191+0000 7fec2b7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:22.209 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.191+0000 7fec2b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:22.209 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.191+0000 7fec2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:22.209 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.191+0000 7fec2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:22.209 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.191+0000 7fec2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:22.209 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.191+0000 7fec2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:22.209 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.191+0000 7fec2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:22.210 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.191+0000 7fec2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:22.210 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.194+0000 7fec2bfff640 10 monclient(hunting): handle_auth_done global_id 4217 payload 824 2023-11-25T05:04:22.210 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.194+0000 7fec2bfff640 1 -- 172.21.15.7:0/3950225591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec2c0a8a00 msgr2=0x7fec2c0ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:22.210 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.194+0000 7fec2bfff640 1 --2- 172.21.15.7:0/3950225591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec2c0a8a00 0x7fec2c0ac3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:22.210 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.194+0000 7fec2bfff640 1 -- 172.21.15.7:0/3950225591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec2c141f20 msgr2=0x7fec2c0ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:22.210 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.194+0000 7fec2bfff640 1 --2- 172.21.15.7:0/3950225591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec2c141f20 0x7fec2c0ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:22.210 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.194+0000 7fec2bfff640 10 monclient: _finish_hunting 0 2023-11-25T05:04:22.210 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.194+0000 7fec2bfff640 1 monclient: found mon.noname-c 2023-11-25T05:04:22.210 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.194+0000 7fec2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:22.210 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.194+0000 7fec2bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:04:22.210 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.194+0000 7fec2bfff640 1 -- 172.21.15.7:0/3950225591 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec20009400 con 0x7fec2c145a40 2023-11-25T05:04:22.210 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.196+0000 7fec2bfff640 1 --2- 172.21.15.7:0/3950225591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fec2c145a40 0x7fec2c1d6f30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fec2400cbf0 tx=0x7fec2400a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:22.211 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.196+0000 7fec13fff640 10 client.?.objecter ms_handle_connect 0x7fec2c145a40 2023-11-25T05:04:22.211 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.196+0000 7fec13fff640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:22.211 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec13fff640 1 -- 172.21.15.7:0/3950225591 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2400ad40 con 0x7fec2c145a40 2023-11-25T05:04:22.211 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:22.211 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec13fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:22.211 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec13fff640 10 monclient: dump: 2023-11-25T05:04:22.211 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:22.211 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:22.211 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:22.211 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:22.211 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:22.211 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:22.212 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:22.212 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:22.212 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:22.212 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:22.212 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec13fff640 10 monclient: _finish_auth 0 2023-11-25T05:04:22.212 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec13fff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:22.212 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:22.212 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec13fff640 1 -- 172.21.15.7:0/3950225591 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec24012400 con 0x7fec2c145a40 2023-11-25T05:04:22.212 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec31d68640 5 monclient: authenticate success, global_id 4217 2023-11-25T05:04:22.212 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec31d68640 10 monclient: _renew_subs 2023-11-25T05:04:22.212 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec31d68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:22.212 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec31d68640 1 -- 172.21.15.7:0/3950225591 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2c1d4790 con 0x7fec2c145a40 2023-11-25T05:04:22.213 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec13fff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:22.213 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec13fff640 1 -- 172.21.15.7:0/3950225591 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2400aea0 con 0x7fec2c145a40 2023-11-25T05:04:22.213 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:22.213 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec13fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:22.213 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec13fff640 10 monclient: dump: 2023-11-25T05:04:22.213 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:22.213 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:22.213 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:22.213 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:22.213 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:22.213 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:22.213 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:22.214 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:22.214 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:22.214 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:22.214 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec29ffb640 4 set_mon_vals no callback set 2023-11-25T05:04:22.214 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec31d68640 10 client.4217.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:22.214 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec31d68640 10 monclient: _renew_subs 2023-11-25T05:04:22.214 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec31d68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:22.214 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec31d68640 1 -- 172.21.15.7:0/3950225591 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2c1d7650 con 0x7fec2c145a40 2023-11-25T05:04:22.214 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.203+0000 7fec31d68640 1 librados: init done 2023-11-25T05:04:22.214 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.204+0000 7fec31d68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:22.214 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.204+0000 7fec31d68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:22.214 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.204+0000 7fec31d68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:22.215 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.204+0000 7fec31d68640 1 -- 172.21.15.7:0/3950225591 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2c0ab850 con 0x7fec2c145a40 2023-11-25T05:04:22.577 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.574+0000 7fec13fff640 1 -- 172.21.15.7:0/3950225591 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7fec24007070 con 0x7fec2c145a40 2023-11-25T05:04:22.577 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.575+0000 7fec13fff640 1 --2- 172.21.15.7:0/3950225591 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fec1806c790 0x7fec1806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:22.577 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.575+0000 7fec2b7fe640 1 --2- 172.21.15.7:0/3950225591 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fec1806c790 0x7fec1806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:22.578 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.575+0000 7fec2b7fe640 10 monclient: get_auth_request con 0x7fec1806c790 auth_method 0 2023-11-25T05:04:22.578 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.575+0000 7fec2b7fe640 1 --2- 172.21.15.7:0/3950225591 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fec1806c790 0x7fec1806ec50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fec1c009b40 tx=0x7fec1c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:22.578 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.576+0000 7fec13fff640 10 client.4217.objecter ms_handle_connect 0x7fec1806c790 2023-11-25T05:04:22.592 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.589+0000 7fec13fff640 1 -- 172.21.15.7:0/3950225591 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2596+0+0 (secure 0 0 0) 0x7fec24058800 con 0x7fec2c145a40 2023-11-25T05:04:22.592 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.589+0000 7fec13fff640 10 client.4217.objecter ms_dispatch 0x7fec2c000b90 osd_map(13..13 src has 1..13) v4 2023-11-25T05:04:22.592 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.589+0000 7fec13fff640 3 client.4217.objecter handle_osd_map got epochs [13,13] > 0 2023-11-25T05:04:22.592 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.589+0000 7fec13fff640 3 client.4217.objecter handle_osd_map decoding full epoch 13 2023-11-25T05:04:22.592 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.589+0000 7fec13fff640 20 client.4217.objecter dump_active .. 0 homeless 2023-11-25T05:04:22.593 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.589+0000 7fec13fff640 1 -- 172.21.15.7:0/3950225591 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec24054b40 con 0x7fec2c145a40 2023-11-25T05:04:22.593 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.589+0000 7fec13fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:22.593 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.589+0000 7fec13fff640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:22.792 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.790+0000 7fec31d68640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:22.792 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.790+0000 7fec31d68640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:22.792 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.790+0000 7fec31d68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:22.793 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:22.790+0000 7fec31d68640 1 -- 172.21.15.7:0/3950225591 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec2c1d6c50 con 0x7fec2c145a40 2023-11-25T05:04:23.135 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.133+0000 7fec13fff640 1 -- 172.21.15.7:0/3950225591 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+9861 (secure 0 0 0) 0x7fec2c1d6c50 con 0x7fec2c145a40 2023-11-25T05:04:23.135 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.133+0000 7fec13fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:23.136 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.133+0000 7fec13fff640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:23.136 INFO:teuthology.misc.health.smithi007.stdout: 2023-11-25T05:04:23.136 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":14,"fsid":"59cf1eff-816d-4eda-b513-a2de7ea218f7","created":"2023-11-25T05:03:35.598582+0000","modified":"2023-11-25T05:04:22.310975+0000","last_up_change":"2023-11-25T05:04:21.270325+0000","last_in_change":"2023-11-25T05:03:41.565581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-25T05:04:21.643061+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":7.1399998664855957,"score_stable":7.1399998664855957,"optimal_score":0.14000000059604645,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5c87807e-7b82-4866-ba36-60b44a25f317","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0e252ab8-9524-4ccc-b08f-d95bbf05fbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccda96b8-aa9e-4c99-8d84-bba19ca0483b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e4b8700a-2e28-4e69-8577-465666b1b051","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"36c88a76-0467-4e37-85b4-dfd5953855a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ebf452a0-1b17-488a-bbea-2e953c932ac7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6811","nonce":132027349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6813","nonce":132027349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6817","nonce":132027349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6815","nonce":132027349}]},"public_addr":"172.21.15.90:6811/132027349","cluster_addr":"172.21.15.90:6813/132027349","heartbeat_back_addr":"172.21.15.90:6817/132027349","heartbeat_front_addr":"172.21.15.90:6815/132027349","state":["exists","up"]},{"osd":6,"uuid":"5aaf35c3-5c06-45e5-b788-8d3207c47cd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d50dd23a-f864-4593-ac00-4d8afed21dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6803","nonce":815268397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6805","nonce":815268397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6809","nonce":815268397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6807","nonce":815268397}]},"public_addr":"172.21.15.90:6803/815268397","cluster_addr":"172.21.15.90:6805/815268397","heartbeat_back_addr":"172.21.15.90:6809/815268397","heartbeat_front_addr":"172.21.15.90:6807/815268397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:19.168168+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:19.564073+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-25T05:04:23.139 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 10 librados: watch_flush enter 2023-11-25T05:04:23.139 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 10 librados: watch_flush exit 2023-11-25T05:04:23.139 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 20 client.4217.objecter shutdown clearing up homeless session... 2023-11-25T05:04:23.139 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 10 client.4217.objecter successfully canceled tick 2023-11-25T05:04:23.139 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 1 -- 172.21.15.7:0/3950225591 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fec1806c790 msgr2=0x7fec1806ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:23.139 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 1 --2- 172.21.15.7:0/3950225591 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fec1806c790 0x7fec1806ec50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fec1c009b40 tx=0x7fec1c009190 comp rx=0 tx=0).stop 2023-11-25T05:04:23.139 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 10 monclient: shutdown 2023-11-25T05:04:23.140 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:23.140 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 1 -- 172.21.15.7:0/3950225591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fec2c145a40 msgr2=0x7fec2c1d6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:23.140 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 1 --2- 172.21.15.7:0/3950225591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fec2c145a40 0x7fec2c1d6f30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fec2400cbf0 tx=0x7fec2400a430 comp rx=0 tx=0).stop 2023-11-25T05:04:23.140 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 1 -- 172.21.15.7:0/3950225591 shutdown_connections 2023-11-25T05:04:23.140 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 1 --2- 172.21.15.7:0/3950225591 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fec1806c790 0x7fec1806ec50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:23.140 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 1 --2- 172.21.15.7:0/3950225591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fec2c145a40 0x7fec2c1d6f30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:23.140 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 1 --2- 172.21.15.7:0/3950225591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec2c141f20 0x7fec2c0ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:23.140 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 1 --2- 172.21.15.7:0/3950225591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec2c0a8a00 0x7fec2c0ac3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:23.140 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 1 -- 172.21.15.7:0/3950225591 >> 172.21.15.7:0/3950225591 conn(0x7fec2c066e80 msgr2=0x7fec2c13e540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:23.140 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 1 -- 172.21.15.7:0/3950225591 shutdown_connections 2023-11-25T05:04:23.140 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 1 -- 172.21.15.7:0/3950225591 wait complete. 2023-11-25T05:04:23.141 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:23.136+0000 7fec31d68640 1 librados: shutdown 2023-11-25T05:04:23.152 DEBUG:teuthology.misc:2 of 8 OSDs are up 2023-11-25T05:04:23.190 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-25T05:04:23.182+0000 ac7d640 -1 osd.0 0 waiting for initial osdmap 2023-11-25T05:04:23.285 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:04:23.271+0000 c493640 -1 osd.2 0 waiting for initial osdmap 2023-11-25T05:04:23.390 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-25T05:04:23.383+0000 c493640 -1 osd.1 0 waiting for initial osdmap 2023-11-25T05:04:23.697 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-25T05:04:23.674+0000 10a53640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-25T05:04:23.766 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:04:23.750+0000 12256640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-25T05:04:23.969 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-25T05:04:23.951+0000 11a55640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-25T05:04:24.146 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-11-25T05:04:24.140+0000 ac7d640 -1 osd.3 0 waiting for initial osdmap 2023-11-25T05:04:24.654 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-11-25T05:04:24.638+0000 11254640 -1 osd.3 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-25T05:04:29.959 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-25T05:04:30.052 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.051+0000 7f0f64bcc640 10 monclient: get_monmap_and_config 2023-11-25T05:04:30.052 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.051+0000 7f0f64bcc640 10 monclient: build_initial_monmap 2023-11-25T05:04:30.052 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.051+0000 7f0f64bcc640 10 monclient: monmap: 2023-11-25T05:04:30.052 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-11-25T05:04:30.053 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:30.053 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:04:30.051975+0000 2023-11-25T05:04:30.053 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:04:30.051975+0000 2023-11-25T05:04:30.053 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:30.053 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:30.053 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:30.053 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:30.053 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:30.053 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:30.053 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:30.053 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:30.055 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.051+0000 7f0f64bcc640 1 Processor -- start 2023-11-25T05:04:30.055 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.051+0000 7f0f64bcc640 1 -- start start 2023-11-25T05:04:30.055 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.051+0000 7f0f64bcc640 10 monclient: init 2023-11-25T05:04:30.055 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.052+0000 7f0f64bcc640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:30.055 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.052+0000 7f0f64bcc640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-11-25T05:04:30.055 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.052+0000 7f0f64bcc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0f600a89e0 0x7f0f6013ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:30.056 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.052+0000 7f0f64bcc640 10 monclient(hunting): picked mon.noname-b con 0x7f0f600a89e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:30.056 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.052+0000 7f0f64bcc640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0f6013f460 0x7f0f601421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:30.056 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.052+0000 7f0f64bcc640 10 monclient(hunting): picked mon.noname-c con 0x7f0f6013f460 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:30.056 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.052+0000 7f0f64bcc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f60142710 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:30.056 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.052+0000 7f0f64bcc640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f600aa770 con 0x7f0f60142710 2023-11-25T05:04:30.056 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.052+0000 7f0f64bcc640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:30.056 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.052+0000 7f0f64bcc640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f6013f860 con 0x7f0f6013f460 2023-11-25T05:04:30.056 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.052+0000 7f0f64bcc640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:30.056 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.052+0000 7f0f64bcc640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f600aa9c0 con 0x7f0f600a89e0 2023-11-25T05:04:30.056 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.052+0000 7f0f64bcc640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:30.056 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.052+0000 7f0f64bcc640 10 monclient(hunting): authenticate will time out at 1396.295439s 2023-11-25T05:04:30.057 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.053+0000 7f0f5e59c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0f600a89e0 0x7f0f6013ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:30.057 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.053+0000 7f0f5e59c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0f600a89e0 0x7f0f6013ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33504/0 (socket says 172.21.15.7:33504) 2023-11-25T05:04:30.057 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.053+0000 7f0f5e59c640 1 -- 172.21.15.7:0/1936935796 learned_addr learned my addr 172.21.15.7:0/1936935796 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:30.057 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.053+0000 7f0f5e59c640 10 monclient(hunting): get_auth_request con 0x7f0f600a89e0 auth_method 0 2023-11-25T05:04:30.057 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.053+0000 7f0f5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:30.057 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.053+0000 7f0f5e59c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:30.057 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.053+0000 7f0f5e59c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:30.057 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.054+0000 7f0f5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:30.057 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.054+0000 7f0f5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:30.057 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.054+0000 7f0f5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:30.058 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.054+0000 7f0f5dd9b640 1 --2- 172.21.15.7:0/1936935796 >> v2:172.21.15.90:3300/0 conn(0x7f0f6013f460 0x7f0f601421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:30.058 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.055+0000 7f0f5dd9b640 10 monclient(hunting): get_auth_request con 0x7f0f6013f460 auth_method 0 2023-11-25T05:04:30.058 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.055+0000 7f0f5dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:30.058 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.055+0000 7f0f5dd9b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:30.059 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.055+0000 7f0f5dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:30.059 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.055+0000 7f0f5e59c640 10 monclient(hunting): handle_auth_done global_id 4226 payload 274 2023-11-25T05:04:30.059 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.055+0000 7f0f5e59c640 1 -- 172.21.15.7:0/1936935796 >> v2:172.21.15.90:3300/0 conn(0x7f0f6013f460 msgr2=0x7f0f601421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:30.059 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.055+0000 7f0f5e59c640 1 --2- 172.21.15.7:0/1936935796 >> v2:172.21.15.90:3300/0 conn(0x7f0f6013f460 0x7f0f601421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.059 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.055+0000 7f0f5e59c640 1 -- 172.21.15.7:0/1936935796 >> v1:172.21.15.90:6789/0 conn(0x7f0f60142710 legacy=0x7f0f60145470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:30.059 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.055+0000 7f0f5e59c640 10 monclient: _finish_hunting 0 2023-11-25T05:04:30.059 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.055+0000 7f0f5e59c640 1 monclient: found mon.noname-b 2023-11-25T05:04:30.059 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.055+0000 7f0f5e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:30.059 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.055+0000 7f0f5e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:04:30.059 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.055+0000 7f0f5e59c640 1 -- 172.21.15.7:0/1936935796 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f600aaca0 con 0x7f0f600a89e0 2023-11-25T05:04:30.061 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.058+0000 7f0f5e59c640 1 --2- 172.21.15.7:0/1936935796 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0f600a89e0 0x7f0f6013ef20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0f50009e50 tx=0x7f0f5002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f57aa8f8ba348a server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:30.063 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f5d59a640 1 -- 172.21.15.7:0/1936935796 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5002ca70 con 0x7f0f600a89e0 2023-11-25T05:04:30.063 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:30.063 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f5d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:30.063 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f5d59a640 10 monclient: dump: 2023-11-25T05:04:30.063 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:30.063 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:30.064 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.064 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.064 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:30.064 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:30.064 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:30.064 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:30.065 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:30.065 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:30.065 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f5d59a640 10 monclient: _finish_auth 0 2023-11-25T05:04:30.065 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f5d59a640 10 monclient: _check_auth_tickets 2023-11-25T05:04:30.065 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:30.065 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f5d59a640 1 -- 172.21.15.7:0/1936935796 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f5002cbd0 con 0x7f0f600a89e0 2023-11-25T05:04:30.065 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f5d59a640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:30.066 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f5d59a640 1 -- 172.21.15.7:0/1936935796 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f500343f0 con 0x7f0f600a89e0 2023-11-25T05:04:30.066 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f64bcc640 5 monclient: authenticate success, global_id 4226 2023-11-25T05:04:30.066 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f64bcc640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:30.066 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f64bcc640 4 set_mon_vals no callback set 2023-11-25T05:04:30.066 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:30.066 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f5d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:30.066 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.061+0000 7f0f5d59a640 10 monclient: dump: 2023-11-25T05:04:30.067 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:30.067 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:30.067 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.067 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.067 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:30.068 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:30.068 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:30.068 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:30.068 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:30.068 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:30.072 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.062+0000 7f0f64bcc640 10 monclient: shutdown 2023-11-25T05:04:30.073 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.062+0000 7f0f64bcc640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:30.073 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.062+0000 7f0f64bcc640 1 -- 172.21.15.7:0/1936935796 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0f600a89e0 msgr2=0x7f0f6013ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:30.073 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.062+0000 7f0f64bcc640 1 --2- 172.21.15.7:0/1936935796 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0f600a89e0 0x7f0f6013ef20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0f50009e50 tx=0x7f0f5002bed0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.073 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.062+0000 7f0f64bcc640 1 -- 172.21.15.7:0/1936935796 shutdown_connections 2023-11-25T05:04:30.073 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.062+0000 7f0f64bcc640 1 --2- 172.21.15.7:0/1936935796 >> v2:172.21.15.90:3300/0 conn(0x7f0f6013f460 0x7f0f601421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.073 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.062+0000 7f0f64bcc640 1 --2- 172.21.15.7:0/1936935796 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0f600a89e0 0x7f0f6013ef20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.073 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.062+0000 7f0f64bcc640 1 -- 172.21.15.7:0/1936935796 >> 172.21.15.7:0/1936935796 conn(0x7f0f600b7940 msgr2=0x7f0f600b7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:30.073 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.062+0000 7f0f64bcc640 1 -- 172.21.15.7:0/1936935796 shutdown_connections 2023-11-25T05:04:30.073 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.062+0000 7f0f64bcc640 1 -- 172.21.15.7:0/1936935796 wait complete. 2023-11-25T05:04:30.073 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.063+0000 7f0f64bcc640 10 monclient: build_initial_monmap 2023-11-25T05:04:30.073 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.063+0000 7f0f64bcc640 10 monclient: monmap: 2023-11-25T05:04:30.073 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-11-25T05:04:30.074 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:30.074 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:30.074 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2023-11-25T05:04:30.074 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:30.074 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:30.074 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:30.074 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:30.074 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:30.074 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:30.074 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.063+0000 7f0f64bcc640 1 librados: starting msgr at 2023-11-25T05:04:30.074 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.063+0000 7f0f64bcc640 1 librados: starting objecter 2023-11-25T05:04:30.074 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.063+0000 7f0f64bcc640 1 Processor -- start 2023-11-25T05:04:30.075 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 1 -- start start 2023-11-25T05:04:30.075 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 1 librados: setting wanted keys 2023-11-25T05:04:30.075 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 1 librados: calling monclient init 2023-11-25T05:04:30.075 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 10 monclient: init 2023-11-25T05:04:30.075 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:30.075 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:04:30.075 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0f600a89e0 0x7f0f6013e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:30.075 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 10 monclient(hunting): picked mon.noname-c con 0x7f0f600a89e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:30.075 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f6013f460 0x7f0f60145ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:30.075 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 10 monclient(hunting): picked mon.noname-b con 0x7f0f6013f460 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:30.075 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f60142710 0x7f0f60143270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:30.075 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 10 monclient(hunting): picked mon.noname-a con 0x7f0f60142710 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:30.076 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:30.076 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f60145600 con 0x7f0f60142710 2023-11-25T05:04:30.076 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:30.076 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f600ac610 con 0x7f0f6013f460 2023-11-25T05:04:30.076 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:30.076 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f601443f0 con 0x7f0f600a89e0 2023-11-25T05:04:30.076 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:30.076 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.064+0000 7f0f64bcc640 10 monclient(hunting): authenticate will time out at 1396.307465s 2023-11-25T05:04:30.076 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5e59c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0f600a89e0 0x7f0f6013e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:30.076 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f6013f460 0x7f0f60145ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:30.076 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f60142710 0x7f0f60143270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:30.076 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5e59c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0f600a89e0 0x7f0f6013e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33516/0 (socket says 172.21.15.7:33516) 2023-11-25T05:04:30.077 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5e59c640 1 -- 172.21.15.7:0/3643125506 learned_addr learned my addr 172.21.15.7:0/3643125506 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:30.077 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5e59c640 10 monclient(hunting): get_auth_request con 0x7f0f600a89e0 auth_method 0 2023-11-25T05:04:30.077 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:30.077 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5e59c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:30.077 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5e59c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:30.077 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5dd9b640 10 monclient(hunting): get_auth_request con 0x7f0f6013f460 auth_method 0 2023-11-25T05:04:30.077 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:30.077 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5dd9b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:30.077 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:30.077 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5ed9d640 10 monclient(hunting): get_auth_request con 0x7f0f60142710 auth_method 0 2023-11-25T05:04:30.077 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:30.077 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5ed9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:30.077 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.065+0000 7f0f5ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:30.078 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.066+0000 7f0f5dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:30.078 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.066+0000 7f0f5dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:30.078 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.066+0000 7f0f5dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:30.078 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.066+0000 7f0f5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:30.078 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.066+0000 7f0f5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:30.078 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.066+0000 7f0f5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:30.078 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.066+0000 7f0f5ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:30.078 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.066+0000 7f0f5ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:30.078 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.066+0000 7f0f5ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:30.078 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.069+0000 7f0f5dd9b640 10 monclient(hunting): handle_auth_done global_id 4264 payload 824 2023-11-25T05:04:30.078 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.069+0000 7f0f5dd9b640 1 -- 172.21.15.7:0/3643125506 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0f600a89e0 msgr2=0x7f0f6013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:30.078 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.069+0000 7f0f5dd9b640 1 --2- 172.21.15.7:0/3643125506 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0f600a89e0 0x7f0f6013e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.079 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.069+0000 7f0f5dd9b640 1 -- 172.21.15.7:0/3643125506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f60142710 msgr2=0x7f0f60143270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:30.079 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.069+0000 7f0f5dd9b640 1 --2- 172.21.15.7:0/3643125506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f60142710 0x7f0f60143270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.079 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.069+0000 7f0f5dd9b640 10 monclient: _finish_hunting 0 2023-11-25T05:04:30.079 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.069+0000 7f0f5dd9b640 1 monclient: found mon.noname-b 2023-11-25T05:04:30.079 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.069+0000 7f0f5dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:30.079 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.069+0000 7f0f5dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:30.079 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.069+0000 7f0f5dd9b640 1 -- 172.21.15.7:0/3643125506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f50009b30 con 0x7f0f6013f460 2023-11-25T05:04:30.079 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.071+0000 7f0f5dd9b640 1 --2- 172.21.15.7:0/3643125506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f6013f460 0x7f0f60145ae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400d5f0 tx=0x7f0f5400dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:30.079 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.071+0000 7f0f4effd640 10 client.?.objecter ms_handle_connect 0x7f0f6013f460 2023-11-25T05:04:30.079 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.071+0000 7f0f4effd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:30.080 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.074+0000 7f0f4effd640 1 -- 172.21.15.7:0/3643125506 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f540125f0 con 0x7f0f6013f460 2023-11-25T05:04:30.081 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.074+0000 7f0f4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:30.081 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.074+0000 7f0f4effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:30.081 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.074+0000 7f0f4effd640 10 monclient: dump: 2023-11-25T05:04:30.081 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:30.081 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:30.081 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.081 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.081 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:30.081 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:30.081 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:30.081 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:30.081 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:30.082 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:30.082 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.074+0000 7f0f4effd640 10 monclient: _finish_auth 0 2023-11-25T05:04:30.082 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.074+0000 7f0f4effd640 10 monclient: _check_auth_tickets 2023-11-25T05:04:30.082 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.074+0000 7f0f4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:30.082 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.074+0000 7f0f4effd640 1 -- 172.21.15.7:0/3643125506 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f54012750 con 0x7f0f6013f460 2023-11-25T05:04:30.082 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.074+0000 7f0f64bcc640 5 monclient: authenticate success, global_id 4264 2023-11-25T05:04:30.082 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.074+0000 7f0f64bcc640 10 monclient: _renew_subs 2023-11-25T05:04:30.082 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f64bcc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:30.082 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f64bcc640 1 -- 172.21.15.7:0/3643125506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f600ab480 con 0x7f0f6013f460 2023-11-25T05:04:30.082 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f4effd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:30.082 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f4effd640 1 -- 172.21.15.7:0/3643125506 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f54012b00 con 0x7f0f6013f460 2023-11-25T05:04:30.082 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:30.082 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:30.083 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f4effd640 10 monclient: dump: 2023-11-25T05:04:30.083 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-11-25T05:04:30.083 INFO:teuthology.misc.health.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:30.083 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.083 INFO:teuthology.misc.health.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.083 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:30.083 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:30.083 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:30.083 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:30.083 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:30.083 INFO:teuthology.misc.health.smithi007.stderr: 2023-11-25T05:04:30.083 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f5cd99640 4 set_mon_vals no callback set 2023-11-25T05:04:30.084 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f64bcc640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:30.084 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f64bcc640 10 monclient: _renew_subs 2023-11-25T05:04:30.084 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f64bcc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:30.084 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f64bcc640 1 -- 172.21.15.7:0/3643125506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f601da8e0 con 0x7f0f6013f460 2023-11-25T05:04:30.084 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f64bcc640 1 librados: init done 2023-11-25T05:04:30.084 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f64bcc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:30.084 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f64bcc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:30.084 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f64bcc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:30.084 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.075+0000 7f0f64bcc640 1 -- 172.21.15.7:0/3643125506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f600b7160 con 0x7f0f6013f460 2023-11-25T05:04:30.317 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.315+0000 7f0f4effd640 1 -- 172.21.15.7:0/3643125506 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f0f54012c60 con 0x7f0f6013f460 2023-11-25T05:04:30.317 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.315+0000 7f0f4effd640 1 --2- 172.21.15.7:0/3643125506 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f0f4006c790 0x7f0f4006ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:30.317 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.316+0000 7f0f5e59c640 1 --2- 172.21.15.7:0/3643125506 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f0f4006c790 0x7f0f4006ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:30.317 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.316+0000 7f0f5e59c640 10 monclient: get_auth_request con 0x7f0f4006c790 auth_method 0 2023-11-25T05:04:30.317 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.316+0000 7f0f5e59c640 1 --2- 172.21.15.7:0/3643125506 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f0f4006c790 0x7f0f4006ec50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0f5002bf10 tx=0x7f0f50030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:30.318 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.316+0000 7f0f4effd640 10 client.4264.objecter ms_handle_connect 0x7f0f4006c790 2023-11-25T05:04:30.319 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.317+0000 7f0f4effd640 1 -- 172.21.15.7:0/3643125506 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0f54059260 con 0x7f0f6013f460 2023-11-25T05:04:30.319 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.317+0000 7f0f4effd640 10 client.4264.objecter ms_dispatch 0x7f0f60000b90 osd_map(18..18 src has 1..18) v4 2023-11-25T05:04:30.319 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.317+0000 7f0f4effd640 3 client.4264.objecter handle_osd_map got epochs [18,18] > 0 2023-11-25T05:04:30.319 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.317+0000 7f0f4effd640 3 client.4264.objecter handle_osd_map decoding full epoch 18 2023-11-25T05:04:30.319 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.317+0000 7f0f4effd640 20 client.4264.objecter dump_active .. 0 homeless 2023-11-25T05:04:30.328 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.327+0000 7f0f4effd640 1 -- 172.21.15.7:0/3643125506 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0f540555a0 con 0x7f0f6013f460 2023-11-25T05:04:30.329 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.327+0000 7f0f4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:30.329 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.327+0000 7f0f4effd640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:30.529 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.527+0000 7f0f64bcc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:30.529 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.527+0000 7f0f64bcc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:30.529 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.527+0000 7f0f64bcc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:30.529 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.527+0000 7f0f64bcc640 1 -- 172.21.15.7:0/3643125506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f600ab010 con 0x7f0f6013f460 2023-11-25T05:04:30.637 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.636+0000 7f0f4effd640 1 -- 172.21.15.7:0/3643125506 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12719 (secure 0 0 0) 0x7f0f54058bc0 con 0x7f0f6013f460 2023-11-25T05:04:30.638 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.636+0000 7f0f4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:30.638 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.636+0000 7f0f4effd640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:30.638 INFO:teuthology.misc.health.smithi007.stdout: 2023-11-25T05:04:30.638 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":18,"fsid":"59cf1eff-816d-4eda-b513-a2de7ea218f7","created":"2023-11-25T05:03:35.598582+0000","modified":"2023-11-25T05:04:26.984346+0000","last_up_change":"2023-11-25T05:04:24.817743+0000","last_in_change":"2023-11-25T05:03:41.565581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-25T05:04:21.643061+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5c87807e-7b82-4866-ba36-60b44a25f317","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6825","nonce":2271997805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6827","nonce":2271997805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6831","nonce":2271997805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6829","nonce":2271997805}]},"public_addr":"172.21.15.7:6825/2271997805","cluster_addr":"172.21.15.7:6827/2271997805","heartbeat_back_addr":"172.21.15.7:6831/2271997805","heartbeat_front_addr":"172.21.15.7:6829/2271997805","state":["exists","up"]},{"osd":1,"uuid":"0e252ab8-9524-4ccc-b08f-d95bbf05fbea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6801","nonce":3792760010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6803","nonce":3792760010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6807","nonce":3792760010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6805","nonce":3792760010}]},"public_addr":"172.21.15.7:6801/3792760010","cluster_addr":"172.21.15.7:6803/3792760010","heartbeat_back_addr":"172.21.15.7:6807/3792760010","heartbeat_front_addr":"172.21.15.7:6805/3792760010","state":["exists","up"]},{"osd":2,"uuid":"ccda96b8-aa9e-4c99-8d84-bba19ca0483b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6809","nonce":1372486177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6811","nonce":1372486177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6815","nonce":1372486177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6813","nonce":1372486177}]},"public_addr":"172.21.15.7:6809/1372486177","cluster_addr":"172.21.15.7:6811/1372486177","heartbeat_back_addr":"172.21.15.7:6815/1372486177","heartbeat_front_addr":"172.21.15.7:6813/1372486177","state":["exists","up"]},{"osd":3,"uuid":"e4b8700a-2e28-4e69-8577-465666b1b051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6817","nonce":59754388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6819","nonce":59754388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6823","nonce":59754388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6821","nonce":59754388}]},"public_addr":"172.21.15.7:6817/59754388","cluster_addr":"172.21.15.7:6819/59754388","heartbeat_back_addr":"172.21.15.7:6823/59754388","heartbeat_front_addr":"172.21.15.7:6821/59754388","state":["exists","up"]},{"osd":4,"uuid":"36c88a76-0467-4e37-85b4-dfd5953855a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6823","nonce":1576115380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6829","nonce":1576115380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6833","nonce":1576115380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6831","nonce":1576115380}]},"public_addr":"172.21.15.90:6823/1576115380","cluster_addr":"172.21.15.90:6829/1576115380","heartbeat_back_addr":"172.21.15.90:6833/1576115380","heartbeat_front_addr":"172.21.15.90:6831/1576115380","state":["exists","up"]},{"osd":5,"uuid":"ebf452a0-1b17-488a-bbea-2e953c932ac7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6811","nonce":132027349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6813","nonce":132027349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6817","nonce":132027349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6815","nonce":132027349}]},"public_addr":"172.21.15.90:6811/132027349","cluster_addr":"172.21.15.90:6813/132027349","heartbeat_back_addr":"172.21.15.90:6817/132027349","heartbeat_front_addr":"172.21.15.90:6815/132027349","state":["exists","up"]},{"osd":6,"uuid":"5aaf35c3-5c06-45e5-b788-8d3207c47cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6819","nonce":3560048676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6821","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6822","nonce":3560048676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6828","nonce":3560048676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6825","nonce":3560048676}]},"public_addr":"172.21.15.90:6819/3560048676","cluster_addr":"172.21.15.90:6822/3560048676","heartbeat_back_addr":"172.21.15.90:6828/3560048676","heartbeat_front_addr":"172.21.15.90:6825/3560048676","state":["exists","up"]},{"osd":7,"uuid":"d50dd23a-f864-4593-ac00-4d8afed21dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6803","nonce":815268397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6805","nonce":815268397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6809","nonce":815268397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6807","nonce":815268397}]},"public_addr":"172.21.15.90:6803/815268397","cluster_addr":"172.21.15.90:6805/815268397","heartbeat_back_addr":"172.21.15.90:6809/815268397","heartbeat_front_addr":"172.21.15.90:6807/815268397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:21.918591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:21.817203+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:21.018791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:22.357339+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:19.776736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:19.168168+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:20.275331+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:19.564073+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-25T05:04:30.640 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 10 librados: watch_flush enter 2023-11-25T05:04:30.640 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 10 librados: watch_flush exit 2023-11-25T05:04:30.640 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 20 client.4264.objecter shutdown clearing up homeless session... 2023-11-25T05:04:30.640 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 10 client.4264.objecter successfully canceled tick 2023-11-25T05:04:30.640 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 1 -- 172.21.15.7:0/3643125506 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f0f4006c790 msgr2=0x7f0f4006ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:30.640 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 1 --2- 172.21.15.7:0/3643125506 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f0f4006c790 0x7f0f4006ec50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0f5002bf10 tx=0x7f0f50030000 comp rx=0 tx=0).stop 2023-11-25T05:04:30.640 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 10 monclient: shutdown 2023-11-25T05:04:30.641 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:30.641 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 1 -- 172.21.15.7:0/3643125506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f6013f460 msgr2=0x7f0f60145ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:30.641 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 1 --2- 172.21.15.7:0/3643125506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f6013f460 0x7f0f60145ae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400d5f0 tx=0x7f0f5400dab0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.641 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 1 -- 172.21.15.7:0/3643125506 shutdown_connections 2023-11-25T05:04:30.641 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 1 --2- 172.21.15.7:0/3643125506 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f0f4006c790 0x7f0f4006ec50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.641 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 1 --2- 172.21.15.7:0/3643125506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f60142710 0x7f0f60143270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.641 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 1 --2- 172.21.15.7:0/3643125506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f6013f460 0x7f0f60145ae0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.641 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 1 --2- 172.21.15.7:0/3643125506 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0f600a89e0 0x7f0f6013e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.641 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 1 -- 172.21.15.7:0/3643125506 >> 172.21.15.7:0/3643125506 conn(0x7f0f600b7940 msgr2=0x7f0f6013c270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:30.641 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 1 -- 172.21.15.7:0/3643125506 shutdown_connections 2023-11-25T05:04:30.641 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 1 -- 172.21.15.7:0/3643125506 wait complete. 2023-11-25T05:04:30.641 INFO:teuthology.misc.health.smithi007.stderr:2023-11-25T05:04:30.638+0000 7f0f64bcc640 1 librados: shutdown 2023-11-25T05:04:30.650 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-11-25T05:04:30.651 INFO:tasks.ceph:Creating RBD pool 2023-11-25T05:04:30.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool create rbd 8 2023-11-25T05:04:30.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.756+0000 7f381dea2640 10 monclient: get_monmap_and_config 2023-11-25T05:04:30.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.756+0000 7f381dea2640 10 monclient: build_initial_monmap 2023-11-25T05:04:30.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.756+0000 7f381dea2640 10 monclient: monmap: 2023-11-25T05:04:30.758 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:30.758 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:30.758 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:30.757143+0000 2023-11-25T05:04:30.758 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:30.757143+0000 2023-11-25T05:04:30.758 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:30.758 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:30.758 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:30.758 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:30.758 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:30.758 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:30.758 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:30.758 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:30.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.756+0000 7f381dea2640 1 Processor -- start 2023-11-25T05:04:30.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 1 -- start start 2023-11-25T05:04:30.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 10 monclient: init 2023-11-25T05:04:30.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:30.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2023-11-25T05:04:30.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f38180a8950 0x7f38180b2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:30.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 10 monclient(hunting): picked mon.noname-a con 0x7f38180a8950 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:30.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38180a8d30 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:30.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f381813f1d0 0x7f381813f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:30.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 10 monclient(hunting): picked mon.noname-c con 0x7f381813f1d0 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:30.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38180a9190 con 0x7f38180a8d30 2023-11-25T05:04:30.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:30.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38180aac00 con 0x7f38180a8950 2023-11-25T05:04:30.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:30.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38181452f0 con 0x7f381813f1d0 2023-11-25T05:04:30.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:30.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.757+0000 7f381dea2640 10 monclient(hunting): authenticate will time out at 1397.000552s 2023-11-25T05:04:30.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.758+0000 7f38177fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f38180a8950 0x7f38180b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:30.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.758+0000 7f3817fff640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f381813f1d0 0x7f381813f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:30.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.758+0000 7f38177fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f38180a8950 0x7f38180b2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40670/0 (socket says 172.21.15.7:40670) 2023-11-25T05:04:30.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.758+0000 7f38177fe640 1 -- 172.21.15.7:0/1361525923 learned_addr learned my addr 172.21.15.7:0/1361525923 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:30.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.758+0000 7f38177fe640 10 monclient(hunting): get_auth_request con 0x7f38180a8950 auth_method 0 2023-11-25T05:04:30.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.758+0000 7f38177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:30.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.758+0000 7f38177fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:30.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.758+0000 7f38177fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:30.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.759+0000 7f3817fff640 10 monclient(hunting): get_auth_request con 0x7f381813f1d0 auth_method 0 2023-11-25T05:04:30.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.759+0000 7f3817fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:30.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.759+0000 7f3817fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:30.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.759+0000 7f3817fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:30.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.759+0000 7f38177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:30.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.759+0000 7f38177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:30.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.759+0000 7f38177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:30.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.759+0000 7f3817fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:30.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.759+0000 7f3817fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:30.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.759+0000 7f3817fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:30.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.761+0000 7f38177fe640 10 monclient(hunting): handle_auth_done global_id 4245 payload 274 2023-11-25T05:04:30.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.761+0000 7f38177fe640 1 -- 172.21.15.7:0/1361525923 >> v2:172.21.15.90:3300/0 conn(0x7f381813f1d0 msgr2=0x7f381813f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:30.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.761+0000 7f38177fe640 1 --2- 172.21.15.7:0/1361525923 >> v2:172.21.15.90:3300/0 conn(0x7f381813f1d0 0x7f381813f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.761+0000 7f38177fe640 1 -- 172.21.15.7:0/1361525923 >> v1:172.21.15.90:6789/0 conn(0x7f38180a8d30 legacy=0x7f3818145a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:30.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.761+0000 7f38177fe640 10 monclient: _finish_hunting 0 2023-11-25T05:04:30.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.761+0000 7f38177fe640 1 monclient: found mon.noname-a 2023-11-25T05:04:30.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.761+0000 7f38177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:30.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.761+0000 7f38177fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:30.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.761+0000 7f38177fe640 1 -- 172.21.15.7:0/1361525923 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38181455d0 con 0x7f38180a8950 2023-11-25T05:04:30.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.763+0000 7f38177fe640 1 --2- 172.21.15.7:0/1361525923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38180a8950 0x7f38180b2180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f380c009870 tx=0x7f380c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e482830ad6f1f11 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:30.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f38167fc640 1 -- 172.21.15.7:0/1361525923 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c031870 con 0x7f38180a8950 2023-11-25T05:04:30.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f38167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:30.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f38167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:30.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f38167fc640 10 monclient: dump: 2023-11-25T05:04:30.769 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:30.769 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:30.769 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.769 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.769 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:30.769 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:30.769 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:30.769 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:30.769 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:30.769 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f38167fc640 10 monclient: _finish_auth 0 2023-11-25T05:04:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f38167fc640 10 monclient: _check_auth_tickets 2023-11-25T05:04:30.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f38167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:30.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f381dea2640 5 monclient: authenticate success, global_id 4245 2023-11-25T05:04:30.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f38167fc640 1 -- 172.21.15.7:0/1361525923 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380c0319d0 con 0x7f38180a8950 2023-11-25T05:04:30.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f38167fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:30.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f381dea2640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:30.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f381dea2640 4 set_mon_vals no callback set 2023-11-25T05:04:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f38167fc640 1 -- 172.21.15.7:0/1361525923 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c031d60 con 0x7f38180a8950 2023-11-25T05:04:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f38167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f38167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f38167fc640 10 monclient: dump: 2023-11-25T05:04:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:30.773 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:30.773 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:30.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f381dea2640 10 monclient: shutdown 2023-11-25T05:04:30.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f381dea2640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:30.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f381dea2640 1 -- 172.21.15.7:0/1361525923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38180a8950 msgr2=0x7f38180b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:30.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f381dea2640 1 --2- 172.21.15.7:0/1361525923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38180a8950 0x7f38180b2180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f380c009870 tx=0x7f380c02b910 comp rx=0 tx=0).stop 2023-11-25T05:04:30.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f381dea2640 1 -- 172.21.15.7:0/1361525923 shutdown_connections 2023-11-25T05:04:30.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f381dea2640 1 --2- 172.21.15.7:0/1361525923 >> v2:172.21.15.90:3300/0 conn(0x7f381813f1d0 0x7f381813f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f381dea2640 1 --2- 172.21.15.7:0/1361525923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38180a8950 0x7f38180b2180 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f381dea2640 1 -- 172.21.15.7:0/1361525923 >> 172.21.15.7:0/1361525923 conn(0x7f3818061900 msgr2=0x7f38180b7780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:30.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f381dea2640 1 -- 172.21.15.7:0/1361525923 shutdown_connections 2023-11-25T05:04:30.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.767+0000 7f381dea2640 1 -- 172.21.15.7:0/1361525923 wait complete. 2023-11-25T05:04:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.768+0000 7f381dea2640 10 monclient: build_initial_monmap 2023-11-25T05:04:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.768+0000 7f381dea2640 10 monclient: monmap: 2023-11-25T05:04:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:30.781 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.768+0000 7f381dea2640 1 librados: starting msgr at 2023-11-25T05:04:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.768+0000 7f381dea2640 1 librados: starting objecter 2023-11-25T05:04:30.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.768+0000 7f381dea2640 1 Processor -- start 2023-11-25T05:04:30.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 1 -- start start 2023-11-25T05:04:30.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 1 librados: setting wanted keys 2023-11-25T05:04:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 1 librados: calling monclient init 2023-11-25T05:04:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 10 monclient: init 2023-11-25T05:04:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:04:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38180a8950 0x7f3818144660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 10 monclient(hunting): picked mon.noname-c con 0x7f38180a8950 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38180a8d30 0x7f381813e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 10 monclient(hunting): picked mon.noname-b con 0x7f38180a8d30 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f381813f1d0 0x7f38181317e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 10 monclient(hunting): picked mon.noname-a con 0x7f381813f1d0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3818144d60 con 0x7f381813f1d0 2023-11-25T05:04:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381813cd80 con 0x7f38180a8d30 2023-11-25T05:04:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381813ea30 con 0x7f38180a8950 2023-11-25T05:04:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.769+0000 7f381dea2640 10 monclient(hunting): authenticate will time out at 1397.012524s 2023-11-25T05:04:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f3817fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f381813f1d0 0x7f38181317e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f38177fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38180a8950 0x7f3818144660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f3816ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38180a8d30 0x7f381813e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f38177fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38180a8950 0x7f3818144660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33532/0 (socket says 172.21.15.7:33532) 2023-11-25T05:04:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f38177fe640 1 -- 172.21.15.7:0/840149659 learned_addr learned my addr 172.21.15.7:0/840149659 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f38177fe640 10 monclient(hunting): get_auth_request con 0x7f38180a8950 auth_method 0 2023-11-25T05:04:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f38177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f38177fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f38177fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f3817fff640 10 monclient(hunting): get_auth_request con 0x7f381813f1d0 auth_method 0 2023-11-25T05:04:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f3817fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f3817fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f3817fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f3816ffd640 10 monclient(hunting): get_auth_request con 0x7f38180a8d30 auth_method 0 2023-11-25T05:04:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f3816ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f3816ffd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.770+0000 7f3816ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.771+0000 7f3816ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:30.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.771+0000 7f3816ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:30.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.771+0000 7f3816ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:30.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.771+0000 7f3817fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:30.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.771+0000 7f3817fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:30.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.771+0000 7f3817fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:30.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.771+0000 7f38177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:30.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.771+0000 7f38177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:30.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.771+0000 7f38177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:30.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.773+0000 7f3816ffd640 10 monclient(hunting): handle_auth_done global_id 4270 payload 824 2023-11-25T05:04:30.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.773+0000 7f3816ffd640 1 -- 172.21.15.7:0/840149659 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38180a8950 msgr2=0x7f3818144660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:30.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.773+0000 7f3816ffd640 1 --2- 172.21.15.7:0/840149659 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38180a8950 0x7f3818144660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.773+0000 7f3816ffd640 1 -- 172.21.15.7:0/840149659 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f381813f1d0 msgr2=0x7f38181317e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:30.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.773+0000 7f3816ffd640 1 --2- 172.21.15.7:0/840149659 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f381813f1d0 0x7f38181317e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:30.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.773+0000 7f3816ffd640 10 monclient: _finish_hunting 0 2023-11-25T05:04:30.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.773+0000 7f3816ffd640 1 monclient: found mon.noname-b 2023-11-25T05:04:30.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.773+0000 7f3816ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:30.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.773+0000 7f3816ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:30.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.773+0000 7f3816ffd640 1 -- 172.21.15.7:0/840149659 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c0094d0 con 0x7f38180a8d30 2023-11-25T05:04:30.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.776+0000 7f3816ffd640 1 --2- 172.21.15.7:0/840149659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38180a8d30 0x7f381813e4f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f380800c390 tx=0x7f380800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:30.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.776+0000 7f37fffff640 10 client.?.objecter ms_handle_connect 0x7f38180a8d30 2023-11-25T05:04:30.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.776+0000 7f37fffff640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:30.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f37fffff640 1 -- 172.21.15.7:0/840149659 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38080052d0 con 0x7f38180a8d30 2023-11-25T05:04:30.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f37fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:30.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f37fffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:30.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f37fffff640 10 monclient: dump: 2023-11-25T05:04:30.791 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:30.791 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:30.791 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.791 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.791 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:30.791 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:30.791 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:30.791 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:30.792 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f37fffff640 10 monclient: _finish_auth 0 2023-11-25T05:04:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f37fffff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f37fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f381dea2640 5 monclient: authenticate success, global_id 4270 2023-11-25T05:04:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f37fffff640 1 -- 172.21.15.7:0/840149659 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3808005430 con 0x7f38180a8d30 2023-11-25T05:04:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f381dea2640 10 monclient: _renew_subs 2023-11-25T05:04:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f381dea2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f381dea2640 1 -- 172.21.15.7:0/840149659 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3818146200 con 0x7f38180a8d30 2023-11-25T05:04:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f37fffff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f37fffff640 1 -- 172.21.15.7:0/840149659 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808003830 con 0x7f38180a8d30 2023-11-25T05:04:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f37fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:30.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f37fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:30.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f37fffff640 10 monclient: dump: 2023-11-25T05:04:30.793 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:30.793 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:30.793 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.793 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:30.793 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:30.793 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:30.793 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:30.793 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:30.793 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:30.793 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:30.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f3815ffb640 4 set_mon_vals no callback set 2023-11-25T05:04:30.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f381dea2640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:30.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f381dea2640 10 monclient: _renew_subs 2023-11-25T05:04:30.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f381dea2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:30.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f381dea2640 1 -- 172.21.15.7:0/840149659 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f381813da50 con 0x7f38180a8d30 2023-11-25T05:04:30.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.779+0000 7f381dea2640 1 librados: init done 2023-11-25T05:04:30.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.780+0000 7f381dea2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:30.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.780+0000 7f381dea2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:30.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.780+0000 7f381dea2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:30.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:30.780+0000 7f381dea2640 1 -- 172.21.15.7:0/840149659 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38180b4040 con 0x7f38180a8d30 2023-11-25T05:04:31.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.018+0000 7f37fffff640 1 -- 172.21.15.7:0/840149659 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f3808003990 con 0x7f38180a8d30 2023-11-25T05:04:31.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.019+0000 7f37fffff640 1 --2- 172.21.15.7:0/840149659 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f380406c790 0x7f380406ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:31.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.019+0000 7f38177fe640 1 --2- 172.21.15.7:0/840149659 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f380406c790 0x7f380406ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:31.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.019+0000 7f38177fe640 10 monclient: get_auth_request con 0x7f380406c790 auth_method 0 2023-11-25T05:04:31.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.019+0000 7f37fffff640 10 client.4270.objecter ms_handle_connect 0x7f380406c790 2023-11-25T05:04:31.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.019+0000 7f38177fe640 1 --2- 172.21.15.7:0/840149659 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f380406c790 0x7f380406ec50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f380c009870 tx=0x7f380c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:31.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.022+0000 7f37fffff640 1 -- 172.21.15.7:0/840149659 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3808057d80 con 0x7f38180a8d30 2023-11-25T05:04:31.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.022+0000 7f37fffff640 10 client.4270.objecter ms_dispatch 0x7f3818000b90 osd_map(18..18 src has 1..18) v4 2023-11-25T05:04:31.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.022+0000 7f37fffff640 3 client.4270.objecter handle_osd_map got epochs [18,18] > 0 2023-11-25T05:04:31.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.022+0000 7f37fffff640 3 client.4270.objecter handle_osd_map decoding full epoch 18 2023-11-25T05:04:31.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.022+0000 7f37fffff640 20 client.4270.objecter dump_active .. 0 homeless 2023-11-25T05:04:31.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.032+0000 7f37fffff640 1 -- 172.21.15.7:0/840149659 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38080540c0 con 0x7f38180a8d30 2023-11-25T05:04:31.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.032+0000 7f37fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:31.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.032+0000 7f37fffff640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:31.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.211+0000 7f381dea2640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-11-25T05:04:31.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.211+0000 7f381dea2640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-11-25T05:04:31.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.211+0000 7f381dea2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:31.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.211+0000 7f381dea2640 1 -- 172.21.15.7:0/840149659 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f3818075b60 con 0x7f38180a8d30 2023-11-25T05:04:31.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.907+0000 7f37fffff640 1 -- 172.21.15.7:0/840149659 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3808054490 con 0x7f38180a8d30 2023-11-25T05:04:31.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.921+0000 7f37fffff640 1 -- 172.21.15.7:0/840149659 <== mon.1 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v19) v1 ==== 109+0+0 (secure 0 0 0) 0x7f380807bd80 con 0x7f38180a8d30 2023-11-25T05:04:31.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.921+0000 7f37fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-11-25T05:04:31.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.921+0000 7f37fffff640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-11-25T05:04:31.923 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'rbd' created 2023-11-25T05:04:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 10 librados: watch_flush enter 2023-11-25T05:04:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 10 librados: watch_flush exit 2023-11-25T05:04:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 20 client.4270.objecter shutdown clearing up homeless session... 2023-11-25T05:04:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 10 client.4270.objecter successfully canceled tick 2023-11-25T05:04:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 1 -- 172.21.15.7:0/840149659 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f380406c790 msgr2=0x7f380406ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 1 --2- 172.21.15.7:0/840149659 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f380406c790 0x7f380406ec50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f380c009870 tx=0x7f380c03c040 comp rx=0 tx=0).stop 2023-11-25T05:04:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 10 monclient: shutdown 2023-11-25T05:04:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 1 -- 172.21.15.7:0/840149659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38180a8d30 msgr2=0x7f381813e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 1 --2- 172.21.15.7:0/840149659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38180a8d30 0x7f381813e4f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f380800c390 tx=0x7f380800c850 comp rx=0 tx=0).stop 2023-11-25T05:04:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 1 -- 172.21.15.7:0/840149659 shutdown_connections 2023-11-25T05:04:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 1 --2- 172.21.15.7:0/840149659 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f380406c790 0x7f380406ec50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 1 --2- 172.21.15.7:0/840149659 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f381813f1d0 0x7f38181317e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 1 --2- 172.21.15.7:0/840149659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38180a8d30 0x7f381813e4f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 1 --2- 172.21.15.7:0/840149659 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38180a8950 0x7f3818144660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 1 -- 172.21.15.7:0/840149659 >> 172.21.15.7:0/840149659 conn(0x7f3818061900 msgr2=0x7f38180b4960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 1 -- 172.21.15.7:0/840149659 shutdown_connections 2023-11-25T05:04:31.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 1 -- 172.21.15.7:0/840149659 wait complete. 2023-11-25T05:04:31.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:31.923+0000 7f381dea2640 1 librados: shutdown 2023-11-25T05:04:31.935 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-25T05:04:32.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.065+0000 7f1450373640 10 monclient: get_monmap_and_config 2023-11-25T05:04:32.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.065+0000 7f1450373640 10 monclient: build_initial_monmap 2023-11-25T05:04:32.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.065+0000 7f1450373640 10 monclient: monmap: 2023-11-25T05:04:32.067 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:32.067 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:32.067 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:32.066356+0000 2023-11-25T05:04:32.067 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:32.066356+0000 2023-11-25T05:04:32.067 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:32.067 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:32.068 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 1 Processor -- start 2023-11-25T05:04:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 1 -- start start 2023-11-25T05:04:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 10 monclient: init 2023-11-25T05:04:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-11-25T05:04:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1448145840 0x7f1448146080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 10 monclient(hunting): picked mon.noname-c con 0x7f1448145840 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14480b20f0 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f14480b2520 0x7f144814c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 10 monclient(hunting): picked mon.noname-b con 0x7f14480b2520 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14480aa850 con 0x7f14480b20f0 2023-11-25T05:04:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14481465c0 con 0x7f1448145840 2023-11-25T05:04:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1448145c20 con 0x7f14480b2520 2023-11-25T05:04:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.066+0000 7f1450373640 10 monclient(hunting): authenticate will time out at 1398.309683s 2023-11-25T05:04:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.067+0000 7f144e910640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f14480b2520 0x7f144814c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.067+0000 7f144e10f640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1448145840 0x7f1448146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.067+0000 7f144d90e640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f14480b20f0 0x7f144813f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37482/0 (socket says 172.21.15.7:37482) 2023-11-25T05:04:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.067+0000 7f144d90e640 1 -- 172.21.15.7:0/127963527 learned_addr learned my addr 172.21.15.7:0/127963527 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.067+0000 7f144e910640 10 monclient(hunting): get_auth_request con 0x7f14480b2520 auth_method 0 2023-11-25T05:04:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.067+0000 7f144e910640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.067+0000 7f144e910640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.067+0000 7f144e910640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.067+0000 7f144e10f640 10 monclient(hunting): get_auth_request con 0x7f1448145840 auth_method 0 2023-11-25T05:04:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.067+0000 7f144e10f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.067+0000 7f144e10f640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.067+0000 7f144e10f640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.068+0000 7f144e10f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.068+0000 7f144e10f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.068+0000 7f144e10f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.068+0000 7f144e910640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.068+0000 7f144e910640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.068+0000 7f144e910640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.070+0000 7f144e10f640 10 monclient(hunting): handle_auth_done global_id 4273 payload 274 2023-11-25T05:04:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.070+0000 7f144e10f640 1 -- 172.21.15.7:0/127963527 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f14480b2520 msgr2=0x7f144814c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.070+0000 7f144e10f640 1 --2- 172.21.15.7:0/127963527 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f14480b2520 0x7f144814c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.070+0000 7f144e10f640 1 -- 172.21.15.7:0/127963527 >> v1:172.21.15.7:6789/0 conn(0x7f14480b20f0 legacy=0x7f144813f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.070+0000 7f144e10f640 10 monclient: _finish_hunting 0 2023-11-25T05:04:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.070+0000 7f144e10f640 1 monclient: found mon.noname-c 2023-11-25T05:04:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.070+0000 7f144e10f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.070+0000 7f144e10f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.070+0000 7f144e10f640 1 -- 172.21.15.7:0/127963527 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14480a9260 con 0x7f1448145840 2023-11-25T05:04:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.072+0000 7f144e10f640 1 --2- 172.21.15.7:0/127963527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1448145840 0x7f1448146080 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f14380097d0 tx=0x7f143802b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5219179e928f6c9 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.074+0000 7f144d10d640 1 -- 172.21.15.7:0/127963527 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1438031770 con 0x7f1448145840 2023-11-25T05:04:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.074+0000 7f144d10d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.074+0000 7f144d10d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.074+0000 7f144d10d640 10 monclient: dump: 2023-11-25T05:04:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:32.077 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.074+0000 7f144d10d640 10 monclient: _finish_auth 0 2023-11-25T05:04:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.074+0000 7f144d10d640 10 monclient: _check_auth_tickets 2023-11-25T05:04:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.074+0000 7f144d10d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.074+0000 7f1450373640 5 monclient: authenticate success, global_id 4273 2023-11-25T05:04:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.074+0000 7f1450373640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.076+0000 7f144d10d640 1 -- 172.21.15.7:0/127963527 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14380318d0 con 0x7f1448145840 2023-11-25T05:04:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.076+0000 7f144d10d640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.076+0000 7f144d10d640 1 -- 172.21.15.7:0/127963527 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1438031c60 con 0x7f1448145840 2023-11-25T05:04:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.076+0000 7f1450373640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.076+0000 7f1450373640 4 set_mon_vals no callback set 2023-11-25T05:04:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.076+0000 7f144d10d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.076+0000 7f144d10d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.076+0000 7f144d10d640 10 monclient: dump: 2023-11-25T05:04:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:32.079 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.077+0000 7f1450373640 10 monclient: shutdown 2023-11-25T05:04:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.077+0000 7f1450373640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.077+0000 7f1450373640 1 -- 172.21.15.7:0/127963527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1448145840 msgr2=0x7f1448146080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.077+0000 7f1450373640 1 --2- 172.21.15.7:0/127963527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1448145840 0x7f1448146080 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f14380097d0 tx=0x7f143802b810 comp rx=0 tx=0).stop 2023-11-25T05:04:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.077+0000 7f1450373640 1 -- 172.21.15.7:0/127963527 shutdown_connections 2023-11-25T05:04:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.077+0000 7f1450373640 1 --2- 172.21.15.7:0/127963527 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f14480b2520 0x7f144814c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.077+0000 7f1450373640 1 --2- 172.21.15.7:0/127963527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1448145840 0x7f1448146080 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.077+0000 7f1450373640 1 -- 172.21.15.7:0/127963527 >> 172.21.15.7:0/127963527 conn(0x7f14480b41f0 msgr2=0x7f14480b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.077+0000 7f1450373640 1 -- 172.21.15.7:0/127963527 shutdown_connections 2023-11-25T05:04:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.077+0000 7f1450373640 1 -- 172.21.15.7:0/127963527 wait complete. 2023-11-25T05:04:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.078+0000 7f1450373640 10 monclient: build_initial_monmap 2023-11-25T05:04:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.078+0000 7f1450373640 10 monclient: monmap: 2023-11-25T05:04:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:32.083 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.078+0000 7f1450373640 1 librados: starting msgr at 2023-11-25T05:04:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.078+0000 7f1450373640 1 librados: starting objecter 2023-11-25T05:04:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.078+0000 7f1450373640 1 Processor -- start 2023-11-25T05:04:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.078+0000 7f1450373640 1 -- start start 2023-11-25T05:04:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.078+0000 7f1450373640 1 librados: setting wanted keys 2023-11-25T05:04:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.078+0000 7f1450373640 1 librados: calling monclient init 2023-11-25T05:04:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.078+0000 7f1450373640 10 monclient: init 2023-11-25T05:04:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-25T05:04:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14480b20f0 0x7f14481dad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 10 monclient(hunting): picked mon.noname-b con 0x7f14480b20f0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14480b2520 0x7f14481cc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 10 monclient(hunting): picked mon.noname-a con 0x7f14480b2520 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1448145840 0x7f144813b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 10 monclient(hunting): picked mon.noname-c con 0x7f1448145840 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14481db2b0 con 0x7f14480b2520 2023-11-25T05:04:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14480ad570 con 0x7f14480b20f0 2023-11-25T05:04:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14481ce230 con 0x7f1448145840 2023-11-25T05:04:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f1450373640 10 monclient(hunting): authenticate will time out at 1398.321937s 2023-11-25T05:04:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f144e10f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14480b20f0 0x7f14481dad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f144e910640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1448145840 0x7f144813b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.079+0000 7f144d90e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14480b2520 0x7f14481cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e10f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14480b20f0 0x7f14481dad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:38232/0 (socket says 172.21.15.7:38232) 2023-11-25T05:04:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e10f640 1 -- 172.21.15.7:0/579866175 learned_addr learned my addr 172.21.15.7:0/579866175 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e10f640 10 monclient(hunting): get_auth_request con 0x7f14480b20f0 auth_method 0 2023-11-25T05:04:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e10f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e10f640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e10f640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e910640 10 monclient(hunting): get_auth_request con 0x7f1448145840 auth_method 0 2023-11-25T05:04:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e910640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e910640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e910640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144d90e640 10 monclient(hunting): get_auth_request con 0x7f14480b2520 auth_method 0 2023-11-25T05:04:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144d90e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144d90e640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144d90e640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e10f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:32.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e10f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:32.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e10f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:32.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e910640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:32.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e910640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:32.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.080+0000 7f144e910640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:32.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.081+0000 7f144d90e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:32.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.081+0000 7f144d90e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:32.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.081+0000 7f144d90e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:32.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.083+0000 7f144e910640 10 monclient(hunting): handle_auth_done global_id 4238 payload 824 2023-11-25T05:04:32.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.083+0000 7f144e910640 1 -- 172.21.15.7:0/579866175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14480b20f0 msgr2=0x7f14481dad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:32.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.083+0000 7f144e910640 1 --2- 172.21.15.7:0/579866175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14480b20f0 0x7f14481dad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:32.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.083+0000 7f144e910640 1 -- 172.21.15.7:0/579866175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14480b2520 msgr2=0x7f14481cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:32.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.083+0000 7f144e910640 1 --2- 172.21.15.7:0/579866175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14480b2520 0x7f14481cc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:32.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.083+0000 7f144e910640 10 monclient: _finish_hunting 0 2023-11-25T05:04:32.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.083+0000 7f144e910640 1 monclient: found mon.noname-c 2023-11-25T05:04:32.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.083+0000 7f144e910640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:32.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.083+0000 7f144e910640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:04:32.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.083+0000 7f144e910640 1 -- 172.21.15.7:0/579866175 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1438009400 con 0x7f1448145840 2023-11-25T05:04:32.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.086+0000 7f144e910640 1 --2- 172.21.15.7:0/579866175 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1448145840 0x7f144813b8b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f144000efc0 tx=0x7f144000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:32.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.086+0000 7f143effd640 10 client.?.objecter ms_handle_connect 0x7f1448145840 2023-11-25T05:04:32.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.086+0000 7f143effd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:32.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.089+0000 7f143effd640 1 -- 172.21.15.7:0/579866175 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1440009c00 con 0x7f1448145840 2023-11-25T05:04:32.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.089+0000 7f143effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:32.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.089+0000 7f143effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:32.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.089+0000 7f143effd640 10 monclient: dump: 2023-11-25T05:04:32.092 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:32.092 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:32.092 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:32.092 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:32.092 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:32.093 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.089+0000 7f143effd640 10 monclient: _finish_auth 0 2023-11-25T05:04:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.089+0000 7f143effd640 10 monclient: _check_auth_tickets 2023-11-25T05:04:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.089+0000 7f143effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.089+0000 7f143effd640 1 -- 172.21.15.7:0/579866175 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144000f040 con 0x7f1448145840 2023-11-25T05:04:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.089+0000 7f1450373640 5 monclient: authenticate success, global_id 4238 2023-11-25T05:04:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.089+0000 7f1450373640 10 monclient: _renew_subs 2023-11-25T05:04:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.089+0000 7f1450373640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.089+0000 7f1450373640 1 -- 172.21.15.7:0/579866175 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144813c570 con 0x7f1448145840 2023-11-25T05:04:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.089+0000 7f143effd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.090+0000 7f143effd640 1 -- 172.21.15.7:0/579866175 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14400146c0 con 0x7f1448145840 2023-11-25T05:04:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.090+0000 7f143effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.090+0000 7f143effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.090+0000 7f143effd640 10 monclient: dump: 2023-11-25T05:04:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:32.095 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:32.095 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:32.095 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:32.095 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:32.095 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:32.095 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:32.095 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:32.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.090+0000 7f144c90c640 4 set_mon_vals no callback set 2023-11-25T05:04:32.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.090+0000 7f1450373640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:32.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.090+0000 7f1450373640 10 monclient: _renew_subs 2023-11-25T05:04:32.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.090+0000 7f1450373640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:32.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.090+0000 7f1450373640 1 -- 172.21.15.7:0/579866175 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14481cefe0 con 0x7f1448145840 2023-11-25T05:04:32.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.090+0000 7f1450373640 1 librados: init done 2023-11-25T05:04:32.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.091+0000 7f1450373640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:32.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.091+0000 7f1450373640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:32.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.091+0000 7f1450373640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:32.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.091+0000 7f1450373640 1 -- 172.21.15.7:0/579866175 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14481db480 con 0x7f1448145840 2023-11-25T05:04:32.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.110+0000 7f143effd640 1 -- 172.21.15.7:0/579866175 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1440014820 con 0x7f1448145840 2023-11-25T05:04:32.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.111+0000 7f143effd640 1 --2- 172.21.15.7:0/579866175 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f142c06c820 0x7f142c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:32.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.111+0000 7f144e10f640 1 --2- 172.21.15.7:0/579866175 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f142c06c820 0x7f142c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:32.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.111+0000 7f144e10f640 10 monclient: get_auth_request con 0x7f142c06c820 auth_method 0 2023-11-25T05:04:32.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.112+0000 7f144e10f640 1 --2- 172.21.15.7:0/579866175 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f142c06c820 0x7f142c06ece0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1438009720 tx=0x7f143803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:32.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.112+0000 7f143effd640 10 client.4238.objecter ms_handle_connect 0x7f142c06c820 2023-11-25T05:04:32.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.378+0000 7f143effd640 1 -- 172.21.15.7:0/579866175 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f1440059600 con 0x7f1448145840 2023-11-25T05:04:32.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.379+0000 7f143effd640 10 client.4238.objecter ms_dispatch 0x7f1448000b90 osd_map(19..19 src has 1..19) v4 2023-11-25T05:04:32.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.379+0000 7f143effd640 3 client.4238.objecter handle_osd_map got epochs [19,19] > 0 2023-11-25T05:04:32.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.379+0000 7f143effd640 3 client.4238.objecter handle_osd_map decoding full epoch 19 2023-11-25T05:04:32.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.379+0000 7f143effd640 20 client.4238.objecter dump_active .. 0 homeless 2023-11-25T05:04:32.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.379+0000 7f143effd640 1 -- 172.21.15.7:0/579866175 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1440055940 con 0x7f1448145840 2023-11-25T05:04:32.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.379+0000 7f143effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:32.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.379+0000 7f143effd640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:32.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.562+0000 7f1450373640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-11-25T05:04:32.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.562+0000 7f1450373640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-11-25T05:04:32.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.562+0000 7f1450373640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:32.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.562+0000 7f1450373640 1 -- 172.21.15.7:0/579866175 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1448067a80 con 0x7f1448145840 2023-11-25T05:04:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.874+0000 7f143effd640 1 -- 172.21.15.7:0/579866175 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v20) v1 ==== 173+0+0 (secure 0 0 0) 0x7f1440018020 con 0x7f1448145840 2023-11-25T05:04:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.874+0000 7f143effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-11-25T05:04:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.874+0000 7f143effd640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-11-25T05:04:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-25T05:04:32.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 10 librados: watch_flush enter 2023-11-25T05:04:32.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 10 librados: watch_flush exit 2023-11-25T05:04:32.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 20 client.4238.objecter shutdown clearing up homeless session... 2023-11-25T05:04:32.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 10 client.4238.objecter successfully canceled tick 2023-11-25T05:04:32.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 1 -- 172.21.15.7:0/579866175 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f142c06c820 msgr2=0x7f142c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:32.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 1 --2- 172.21.15.7:0/579866175 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f142c06c820 0x7f142c06ece0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1438009720 tx=0x7f143803d040 comp rx=0 tx=0).stop 2023-11-25T05:04:32.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 10 monclient: shutdown 2023-11-25T05:04:32.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:32.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 1 -- 172.21.15.7:0/579866175 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1448145840 msgr2=0x7f144813b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:32.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 1 --2- 172.21.15.7:0/579866175 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1448145840 0x7f144813b8b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f144000efc0 tx=0x7f144000c490 comp rx=0 tx=0).stop 2023-11-25T05:04:32.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 1 -- 172.21.15.7:0/579866175 shutdown_connections 2023-11-25T05:04:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 1 --2- 172.21.15.7:0/579866175 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f142c06c820 0x7f142c06ece0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 1 --2- 172.21.15.7:0/579866175 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1448145840 0x7f144813b8b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 1 --2- 172.21.15.7:0/579866175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14480b2520 0x7f14481cc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 1 --2- 172.21.15.7:0/579866175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14480b20f0 0x7f14481dad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.876+0000 7f1450373640 1 -- 172.21.15.7:0/579866175 >> 172.21.15.7:0/579866175 conn(0x7f14480b41f0 msgr2=0x7f1448144d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.877+0000 7f1450373640 1 -- 172.21.15.7:0/579866175 shutdown_connections 2023-11-25T05:04:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.877+0000 7f1450373640 1 -- 172.21.15.7:0/579866175 wait complete. 2023-11-25T05:04:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:32.877+0000 7f1450373640 1 librados: shutdown 2023-11-25T05:04:32.889 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-25T05:04:32.890 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-25T05:04:32.890 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-25T05:04:32.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-25T05:04:32.891 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-25T05:04:33.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.017+0000 7f3be7789640 10 monclient: get_monmap_and_config 2023-11-25T05:04:33.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.017+0000 7f3be7789640 10 monclient: build_initial_monmap 2023-11-25T05:04:33.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.017+0000 7f3be7789640 10 monclient: monmap: 2023-11-25T05:04:33.019 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:33.019 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:33.019 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:33.018543+0000 2023-11-25T05:04:33.019 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:33.018543+0000 2023-11-25T05:04:33.019 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:33.019 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:33.020 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:33.020 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:33.020 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:33.020 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:33.020 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:33.020 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:33.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.018+0000 7f3be7789640 1 Processor -- start 2023-11-25T05:04:33.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.018+0000 7f3be7789640 1 -- start start 2023-11-25T05:04:33.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.018+0000 7f3be7789640 10 monclient: init 2023-11-25T05:04:33.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.018+0000 7f3be7789640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:33.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.018+0000 7f3be7789640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2023-11-25T05:04:33.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.018+0000 7f3be7789640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3be00a8910 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:33.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.018+0000 7f3be7789640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3be00a9b70 0x7f3be013f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:33.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.019+0000 7f3be7789640 10 monclient(hunting): picked mon.noname-c con 0x7f3be00a9b70 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:33.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.019+0000 7f3be7789640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3be0141f90 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:33.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.019+0000 7f3be7789640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be00a8cf0 con 0x7f3be0141f90 2023-11-25T05:04:33.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.019+0000 7f3be7789640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be00aabd0 con 0x7f3be00a8910 2023-11-25T05:04:33.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.019+0000 7f3be7789640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:33.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.019+0000 7f3be7789640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3be00aaeb0 con 0x7f3be00a9b70 2023-11-25T05:04:33.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.019+0000 7f3be7789640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:33.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.019+0000 7f3be7789640 10 monclient(hunting): authenticate will time out at 1399.261894s 2023-11-25T05:04:33.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.019+0000 7f3be4d24640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3be00a9b70 0x7f3be013f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:33.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.020+0000 7f3be4d24640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3be00a9b70 0x7f3be013f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:38240/0 (socket says 172.21.15.7:38240) 2023-11-25T05:04:33.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.020+0000 7f3be4d24640 1 -- 172.21.15.7:0/1160435332 learned_addr learned my addr 172.21.15.7:0/1160435332 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:33.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.020+0000 7f3be4d24640 10 monclient(hunting): get_auth_request con 0x7f3be00a9b70 auth_method 0 2023-11-25T05:04:33.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.020+0000 7f3be4d24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:33.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.020+0000 7f3be4d24640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:33.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.020+0000 7f3be4d24640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:33.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.021+0000 7f3be4d24640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:33.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.021+0000 7f3be4d24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:33.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.021+0000 7f3be4d24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:33.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.024+0000 7f3be4d24640 10 monclient(hunting): handle_auth_done global_id 4279 payload 274 2023-11-25T05:04:33.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.024+0000 7f3be4d24640 1 -- 172.21.15.7:0/1160435332 >> v1:172.21.15.90:6789/0 conn(0x7f3be00a8910 legacy=0x7f3be013ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:33.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.024+0000 7f3be4d24640 1 -- 172.21.15.7:0/1160435332 >> v1:172.21.15.7:6789/0 conn(0x7f3be0141f90 legacy=0x7f3be01423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:33.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.024+0000 7f3be4d24640 10 monclient: _finish_hunting 0 2023-11-25T05:04:33.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.024+0000 7f3be4d24640 1 monclient: found mon.noname-c 2023-11-25T05:04:33.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.024+0000 7f3be4d24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:33.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.024+0000 7f3be4d24640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:33.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.024+0000 7f3be4d24640 1 -- 172.21.15.7:0/1160435332 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be00a9720 con 0x7f3be00a9b70 2023-11-25T05:04:33.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.030+0000 7f3be4d24640 1 --2- 172.21.15.7:0/1160435332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3be00a9b70 0x7f3be013f370 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0009870 tx=0x7f3bd002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de6b5a21ba2ae877 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:33.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3bd7fff640 1 -- 172.21.15.7:0/1160435332 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0030860 con 0x7f3be00a9b70 2023-11-25T05:04:33.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3bd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:33.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3bd7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:33.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3bd7fff640 10 monclient: dump: 2023-11-25T05:04:33.038 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:33.038 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:33.038 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:33.038 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:33.038 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:33.038 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:33.039 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:33.039 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:33.039 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:33.039 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:33.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3bd7fff640 10 monclient: _finish_auth 0 2023-11-25T05:04:33.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3bd7fff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:33.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3bd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:33.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3bd7fff640 1 -- 172.21.15.7:0/1160435332 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd00309c0 con 0x7f3be00a9b70 2023-11-25T05:04:33.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3be7789640 5 monclient: authenticate success, global_id 4279 2023-11-25T05:04:33.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3be7789640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:33.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3bd7fff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:33.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3be7789640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:33.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3be7789640 4 set_mon_vals no callback set 2023-11-25T05:04:33.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3bd7fff640 1 -- 172.21.15.7:0/1160435332 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0030d50 con 0x7f3be00a9b70 2023-11-25T05:04:33.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3bd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:33.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3bd7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:33.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.036+0000 7f3bd7fff640 10 monclient: dump: 2023-11-25T05:04:33.040 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:33.040 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:33.040 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:33.041 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:33.041 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:33.041 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:33.041 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:33.041 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:33.041 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:33.041 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:33.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.037+0000 7f3be7789640 10 monclient: shutdown 2023-11-25T05:04:33.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.037+0000 7f3be7789640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:33.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.037+0000 7f3be7789640 1 -- 172.21.15.7:0/1160435332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3be00a9b70 msgr2=0x7f3be013f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:33.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.037+0000 7f3be7789640 1 --2- 172.21.15.7:0/1160435332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3be00a9b70 0x7f3be013f370 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0009870 tx=0x7f3bd002b9a0 comp rx=0 tx=0).stop 2023-11-25T05:04:33.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.037+0000 7f3be7789640 1 -- 172.21.15.7:0/1160435332 shutdown_connections 2023-11-25T05:04:33.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.037+0000 7f3be7789640 1 --2- 172.21.15.7:0/1160435332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3be00a9b70 0x7f3be013f370 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:33.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.037+0000 7f3be7789640 1 -- 172.21.15.7:0/1160435332 >> 172.21.15.7:0/1160435332 conn(0x7f3be00b7780 msgr2=0x7f3be00b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.037+0000 7f3be7789640 1 -- 172.21.15.7:0/1160435332 shutdown_connections 2023-11-25T05:04:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.037+0000 7f3be7789640 1 -- 172.21.15.7:0/1160435332 wait complete. 2023-11-25T05:04:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.038+0000 7f3be7789640 10 monclient: build_initial_monmap 2023-11-25T05:04:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.038+0000 7f3be7789640 10 monclient: monmap: 2023-11-25T05:04:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:33.045 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:33.046 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:33.046 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:33.046 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:33.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.038+0000 7f3be7789640 1 librados: starting msgr at 2023-11-25T05:04:33.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.038+0000 7f3be7789640 1 librados: starting objecter 2023-11-25T05:04:33.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.038+0000 7f3be7789640 1 Processor -- start 2023-11-25T05:04:33.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 1 -- start start 2023-11-25T05:04:33.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 1 librados: setting wanted keys 2023-11-25T05:04:33.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 1 librados: calling monclient init 2023-11-25T05:04:33.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 10 monclient: init 2023-11-25T05:04:33.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:33.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-25T05:04:33.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3be00a8910 0x7f3be0138140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:33.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 10 monclient(hunting): picked mon.noname-c con 0x7f3be00a8910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:33.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be0141f90 0x7f3be013afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:33.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 10 monclient(hunting): picked mon.noname-a con 0x7f3be0141f90 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:33.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3be0139f80 0x7f3be0141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:33.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 10 monclient(hunting): picked mon.noname-b con 0x7f3be0139f80 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:33.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be01367d0 con 0x7f3be0141f90 2023-11-25T05:04:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be0135af0 con 0x7f3be0139f80 2023-11-25T05:04:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be0135d90 con 0x7f3be00a8910 2023-11-25T05:04:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.039+0000 7f3be7789640 10 monclient(hunting): authenticate will time out at 1399.282764s 2023-11-25T05:04:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.040+0000 7f3be5525640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3be00a8910 0x7f3be0138140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.040+0000 7f3be5d26640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3be0139f80 0x7f3be0141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.040+0000 7f3be5525640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3be00a8910 0x7f3be0138140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33546/0 (socket says 172.21.15.7:33546) 2023-11-25T05:04:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.040+0000 7f3be5525640 1 -- 172.21.15.7:0/786638581 learned_addr learned my addr 172.21.15.7:0/786638581 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.040+0000 7f3be5525640 10 monclient(hunting): get_auth_request con 0x7f3be00a8910 auth_method 0 2023-11-25T05:04:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.040+0000 7f3be5525640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.040+0000 7f3be5525640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.040+0000 7f3be5525640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.041+0000 7f3be5d26640 10 monclient(hunting): get_auth_request con 0x7f3be0139f80 auth_method 0 2023-11-25T05:04:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.041+0000 7f3be5d26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.041+0000 7f3be5d26640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.041+0000 7f3be5d26640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.041+0000 7f3be5525640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.041+0000 7f3be5525640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.041+0000 7f3be5525640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.041+0000 7f3be5d26640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.041+0000 7f3be5d26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.041+0000 7f3be5d26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.044+0000 7f3be5525640 10 monclient(hunting): handle_auth_done global_id 4241 payload 824 2023-11-25T05:04:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.044+0000 7f3be5525640 1 -- 172.21.15.7:0/786638581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3be0139f80 msgr2=0x7f3be0141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:33.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.044+0000 7f3be5525640 1 --2- 172.21.15.7:0/786638581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3be0139f80 0x7f3be0141400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:33.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.044+0000 7f3be5525640 1 -- 172.21.15.7:0/786638581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be0141f90 msgr2=0x7f3be013afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:33.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.044+0000 7f3be5525640 1 --2- 172.21.15.7:0/786638581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be0141f90 0x7f3be013afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:33.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.044+0000 7f3be5525640 10 monclient: _finish_hunting 0 2023-11-25T05:04:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.044+0000 7f3be5525640 1 monclient: found mon.noname-c 2023-11-25T05:04:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.044+0000 7f3be5525640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.044+0000 7f3be5525640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:04:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.044+0000 7f3be5525640 1 -- 172.21.15.7:0/786638581 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd00094d0 con 0x7f3be00a8910 2023-11-25T05:04:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.047+0000 7f3be5525640 1 --2- 172.21.15.7:0/786638581 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3be00a8910 0x7f3be0138140 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc00c3c0 tx=0x7f3bdc00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.047+0000 7f3bd5ffb640 10 client.?.objecter ms_handle_connect 0x7f3be00a8910 2023-11-25T05:04:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.047+0000 7f3bd5ffb640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3bd5ffb640 1 -- 172.21.15.7:0/786638581 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc0052d0 con 0x7f3be00a8910 2023-11-25T05:04:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3bd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3bd5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3bd5ffb640 10 monclient: dump: 2023-11-25T05:04:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:33.062 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3bd5ffb640 10 monclient: _finish_auth 0 2023-11-25T05:04:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3bd5ffb640 10 monclient: _check_auth_tickets 2023-11-25T05:04:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3bd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3bd5ffb640 1 -- 172.21.15.7:0/786638581 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bdc005450 con 0x7f3be00a8910 2023-11-25T05:04:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3be7789640 5 monclient: authenticate success, global_id 4241 2023-11-25T05:04:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3bd5ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3bd5ffb640 1 -- 172.21.15.7:0/786638581 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc007260 con 0x7f3be00a8910 2023-11-25T05:04:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3be7789640 10 monclient: _renew_subs 2023-11-25T05:04:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3be7789640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3be7789640 1 -- 172.21.15.7:0/786638581 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be0141940 con 0x7f3be00a8910 2023-11-25T05:04:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3bd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3bd77fe640 4 set_mon_vals no callback set 2023-11-25T05:04:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3bd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3bd5ffb640 10 monclient: dump: 2023-11-25T05:04:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:33.064 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:33.064 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:33.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3be7789640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:33.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3be7789640 10 monclient: _renew_subs 2023-11-25T05:04:33.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3be7789640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:33.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3be7789640 1 -- 172.21.15.7:0/786638581 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be0130470 con 0x7f3be00a8910 2023-11-25T05:04:33.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.051+0000 7f3be7789640 1 librados: init done 2023-11-25T05:04:33.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.052+0000 7f3be7789640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:33.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.052+0000 7f3be7789640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:33.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.052+0000 7f3be7789640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:33.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.052+0000 7f3be7789640 1 -- 172.21.15.7:0/786638581 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be006ace0 con 0x7f3be00a8910 2023-11-25T05:04:33.082 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:15.280 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:33.160 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:15.358 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:33.285 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:15.336 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:33.285 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:15.411 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:33.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.388+0000 7f3bd5ffb640 1 -- 172.21.15.7:0/786638581 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3bdc0073c0 con 0x7f3be00a8910 2023-11-25T05:04:33.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.389+0000 7f3bd5ffb640 1 --2- 172.21.15.7:0/786638581 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f3bc406c7d0 0x7f3bc406ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:33.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.389+0000 7f3be4d24640 1 --2- 172.21.15.7:0/786638581 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f3bc406c7d0 0x7f3bc406ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:33.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.389+0000 7f3be4d24640 10 monclient: get_auth_request con 0x7f3bc406c7d0 auth_method 0 2023-11-25T05:04:33.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.390+0000 7f3bd5ffb640 10 client.4241.objecter ms_handle_connect 0x7f3bc406c7d0 2023-11-25T05:04:33.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.390+0000 7f3be4d24640 1 --2- 172.21.15.7:0/786638581 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f3bc406c7d0 0x7f3bc406ec90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0036000 tx=0x7f3bd003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.414+0000 7f3bd5ffb640 1 -- 172.21.15.7:0/786638581 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3bdc057cb0 con 0x7f3be00a8910 2023-11-25T05:04:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.414+0000 7f3bd5ffb640 10 client.4241.objecter ms_dispatch 0x7f3be0000b90 osd_map(20..20 src has 1..20) v4 2023-11-25T05:04:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.414+0000 7f3bd5ffb640 3 client.4241.objecter handle_osd_map got epochs [20,20] > 0 2023-11-25T05:04:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.414+0000 7f3bd5ffb640 3 client.4241.objecter handle_osd_map decoding full epoch 20 2023-11-25T05:04:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.414+0000 7f3bd5ffb640 20 client.4241.objecter dump_active .. 0 homeless 2023-11-25T05:04:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.414+0000 7f3bd5ffb640 1 -- 172.21.15.7:0/786638581 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3bdc053ff0 con 0x7f3be00a8910 2023-11-25T05:04:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.414+0000 7f3bd5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:33.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.414+0000 7f3bd5ffb640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:33.442 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:15.640 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:33.529 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:15.673 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:33.542 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:15.740 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:33.603 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:15.801 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.785+0000 7f3be7789640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-11-25T05:04:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.785+0000 7f3be7789640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-11-25T05:04:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.785+0000 7f3be7789640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:33.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:33.785+0000 7f3be7789640 1 -- 172.21.15.7:0/786638581 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f3be0075b60 con 0x7f3be00a8910 2023-11-25T05:04:33.960 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:16.158 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:34.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.262+0000 7f3bd5ffb640 1 -- 172.21.15.7:0/786638581 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171031 (secure 0 0 0) 0x7f3bdc020020 con 0x7f3be00a8910 2023-11-25T05:04:34.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.262+0000 7f3bd5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-11-25T05:04:34.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.262+0000 7f3bd5ffb640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:34.267 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:04:34.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 10 librados: watch_flush enter 2023-11-25T05:04:34.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 10 librados: watch_flush exit 2023-11-25T05:04:34.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 20 client.4241.objecter shutdown clearing up homeless session... 2023-11-25T05:04:34.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 10 client.4241.objecter successfully canceled tick 2023-11-25T05:04:34.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 1 -- 172.21.15.7:0/786638581 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f3bc406c7d0 msgr2=0x7f3bc406ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 1 --2- 172.21.15.7:0/786638581 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f3bc406c7d0 0x7f3bc406ec90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0036000 tx=0x7f3bd003b040 comp rx=0 tx=0).stop 2023-11-25T05:04:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 10 monclient: shutdown 2023-11-25T05:04:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 1 -- 172.21.15.7:0/786638581 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3be00a8910 msgr2=0x7f3be0138140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 1 --2- 172.21.15.7:0/786638581 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3be00a8910 0x7f3be0138140 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc00c3c0 tx=0x7f3bdc00c880 comp rx=0 tx=0).stop 2023-11-25T05:04:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 1 -- 172.21.15.7:0/786638581 shutdown_connections 2023-11-25T05:04:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 1 --2- 172.21.15.7:0/786638581 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f3bc406c7d0 0x7f3bc406ec90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 1 --2- 172.21.15.7:0/786638581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3be0139f80 0x7f3be0141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 1 --2- 172.21.15.7:0/786638581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be0141f90 0x7f3be013afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 1 --2- 172.21.15.7:0/786638581 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3be00a8910 0x7f3be0138140 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 1 -- 172.21.15.7:0/786638581 >> 172.21.15.7:0/786638581 conn(0x7f3be00b7780 msgr2=0x7f3be013f180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 1 -- 172.21.15.7:0/786638581 shutdown_connections 2023-11-25T05:04:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 1 -- 172.21.15.7:0/786638581 wait complete. 2023-11-25T05:04:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.265+0000 7f3be7789640 1 librados: shutdown 2023-11-25T05:04:34.278 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":5,"active_gid":4110,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":1055092586},{"type":"v1","addr":"172.21.15.90:6801","nonce":1055092586}]},"active_addr":"172.21.15.90:6801/1055092586","active_change":"2023-11-25T05:03:41.188240+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4102,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":3083209156}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":2022433166}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":2194197068}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":257799453}]}]} 2023-11-25T05:04:34.284 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-25T05:04:34.284 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-25T05:04:34.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-25T05:04:34.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.399+0000 7f2e06a4b640 10 monclient: get_monmap_and_config 2023-11-25T05:04:34.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.399+0000 7f2e06a4b640 10 monclient: build_initial_monmap 2023-11-25T05:04:34.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.399+0000 7f2e06a4b640 10 monclient: monmap: 2023-11-25T05:04:34.401 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:34.401 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:34.401 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:34.400401+0000 2023-11-25T05:04:34.401 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:34.400401+0000 2023-11-25T05:04:34.401 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:34.401 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:34.402 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:34.402 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:34.402 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:34.402 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:34.402 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:34.402 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:34.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.400+0000 7f2e06a4b640 1 Processor -- start 2023-11-25T05:04:34.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.400+0000 7f2e06a4b640 1 -- start start 2023-11-25T05:04:34.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.400+0000 7f2e06a4b640 10 monclient: init 2023-11-25T05:04:34.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.400+0000 7f2e06a4b640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:34.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.400+0000 7f2e06a4b640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-11-25T05:04:34.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2e06a4b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2e000a8a30 0x7f2e001424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:34.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2e06a4b640 10 monclient(hunting): picked mon.noname-a con 0x7f2e000a8a30 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:34.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2e06a4b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e000a8e10 0x7f2e00142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:34.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2e06a4b640 10 monclient(hunting): picked mon.noname-b con 0x7f2e000a8e10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:34.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2e06a4b640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2e001454a0 0x7f2e00145930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:34.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2e06a4b640 10 monclient(hunting): picked mon.noname-c con 0x7f2e001454a0 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:34.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2e06a4b640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:34.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2e06a4b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e000aaa10 con 0x7f2e000a8a30 2023-11-25T05:04:34.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2e06a4b640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:34.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2e06a4b640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e000aac10 con 0x7f2e001454a0 2023-11-25T05:04:34.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2e06a4b640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:34.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2e06a4b640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e000aaef0 con 0x7f2e000a8e10 2023-11-25T05:04:34.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2e06a4b640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:34.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2e06a4b640 10 monclient(hunting): authenticate will time out at 1400.643966s 2023-11-25T05:04:34.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2dfffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2e000a8a30 0x7f2e001424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:34.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.401+0000 7f2e04fe8640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2e001454a0 0x7f2e00145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:34.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2dfffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2e000a8a30 0x7f2e001424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40714/0 (socket says 172.21.15.7:40714) 2023-11-25T05:04:34.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2dfffff640 1 -- 172.21.15.7:0/3223105260 learned_addr learned my addr 172.21.15.7:0/3223105260 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:34.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2dfffff640 10 monclient(hunting): get_auth_request con 0x7f2e000a8a30 auth_method 0 2023-11-25T05:04:34.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2dfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:34.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2dfffff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:34.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2dfffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:34.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2e04fe8640 10 monclient(hunting): get_auth_request con 0x7f2e001454a0 auth_method 0 2023-11-25T05:04:34.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2e04fe8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:34.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2e04fe8640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:34.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2e04fe8640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:34.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2e04fe8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:34.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2e04fe8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:34.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2e04fe8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:34.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2dfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:34.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2dfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:34.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.402+0000 7f2dfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:34.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.404+0000 7f2e04fe8640 10 monclient(hunting): handle_auth_done global_id 4285 payload 274 2023-11-25T05:04:34.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.404+0000 7f2e04fe8640 1 -- 172.21.15.7:0/3223105260 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e000a8e10 msgr2=0x7f2e00142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:34.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.404+0000 7f2e04fe8640 1 --2- 172.21.15.7:0/3223105260 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e000a8e10 0x7f2e00142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.404+0000 7f2e04fe8640 1 -- 172.21.15.7:0/3223105260 >> v2:172.21.15.7:3300/0 conn(0x7f2e000a8a30 msgr2=0x7f2e001424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:34.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.404+0000 7f2e04fe8640 1 --2- 172.21.15.7:0/3223105260 >> v2:172.21.15.7:3300/0 conn(0x7f2e000a8a30 0x7f2e001424e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.404+0000 7f2e04fe8640 10 monclient: _finish_hunting 0 2023-11-25T05:04:34.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.404+0000 7f2e04fe8640 1 monclient: found mon.noname-c 2023-11-25T05:04:34.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.404+0000 7f2e04fe8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:34.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.404+0000 7f2e04fe8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:34.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.404+0000 7f2e04fe8640 1 -- 172.21.15.7:0/3223105260 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e000a97f0 con 0x7f2e001454a0 2023-11-25T05:04:34.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.406+0000 7f2e04fe8640 1 --2- 172.21.15.7:0/3223105260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e001454a0 0x7f2e00145930 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2df8009d90 tx=0x7f2df802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8639ab444f7e001 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:34.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2dfeffd640 1 -- 172.21.15.7:0/3223105260 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df802cbc0 con 0x7f2e001454a0 2023-11-25T05:04:34.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2dfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:34.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2dfeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:34.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2dfeffd640 10 monclient: dump: 2023-11-25T05:04:34.416 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:34.416 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:34.416 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:34.417 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2dfeffd640 10 monclient: _finish_auth 0 2023-11-25T05:04:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2dfeffd640 10 monclient: _check_auth_tickets 2023-11-25T05:04:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2dfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2dfeffd640 1 -- 172.21.15.7:0/3223105260 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df802cd20 con 0x7f2e001454a0 2023-11-25T05:04:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2e06a4b640 5 monclient: authenticate success, global_id 4285 2023-11-25T05:04:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2e06a4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2dfeffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2e06a4b640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2e06a4b640 4 set_mon_vals no callback set 2023-11-25T05:04:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2dfeffd640 1 -- 172.21.15.7:0/3223105260 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df8034660 con 0x7f2e001454a0 2023-11-25T05:04:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2dfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2dfeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2dfeffd640 10 monclient: dump: 2023-11-25T05:04:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:34.419 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2e06a4b640 10 monclient: shutdown 2023-11-25T05:04:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2e06a4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2e06a4b640 1 -- 172.21.15.7:0/3223105260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e001454a0 msgr2=0x7f2e00145930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2e06a4b640 1 --2- 172.21.15.7:0/3223105260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e001454a0 0x7f2e00145930 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2df8009d90 tx=0x7f2df802be10 comp rx=0 tx=0).stop 2023-11-25T05:04:34.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2e06a4b640 1 -- 172.21.15.7:0/3223105260 shutdown_connections 2023-11-25T05:04:34.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2e06a4b640 1 --2- 172.21.15.7:0/3223105260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e001454a0 0x7f2e00145930 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2e06a4b640 1 --2- 172.21.15.7:0/3223105260 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e000a8e10 0x7f2e00142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2e06a4b640 1 --2- 172.21.15.7:0/3223105260 >> v2:172.21.15.7:3300/0 conn(0x7f2e000a8a30 0x7f2e001424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2e06a4b640 1 -- 172.21.15.7:0/3223105260 >> 172.21.15.7:0/3223105260 conn(0x7f2e00061a30 msgr2=0x7f2e000b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:34.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2e06a4b640 1 -- 172.21.15.7:0/3223105260 shutdown_connections 2023-11-25T05:04:34.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.410+0000 7f2e06a4b640 1 -- 172.21.15.7:0/3223105260 wait complete. 2023-11-25T05:04:34.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.411+0000 7f2e06a4b640 10 monclient: build_initial_monmap 2023-11-25T05:04:34.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.411+0000 7f2e06a4b640 10 monclient: monmap: 2023-11-25T05:04:34.421 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:34.421 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:34.421 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:34.421 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:34.421 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:34.422 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.411+0000 7f2e06a4b640 1 librados: starting msgr at 2023-11-25T05:04:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.411+0000 7f2e06a4b640 1 librados: starting objecter 2023-11-25T05:04:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.411+0000 7f2e06a4b640 1 Processor -- start 2023-11-25T05:04:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 1 -- start start 2023-11-25T05:04:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 1 librados: setting wanted keys 2023-11-25T05:04:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 1 librados: calling monclient init 2023-11-25T05:04:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 10 monclient: init 2023-11-25T05:04:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:04:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e000a8a30 0x7f2e000abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 10 monclient(hunting): picked mon.noname-c con 0x7f2e000a8a30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e000a8e10 0x7f2e000ac9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 10 monclient(hunting): picked mon.noname-b con 0x7f2e000a8e10 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e001454a0 0x7f2e001d7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 10 monclient(hunting): picked mon.noname-a con 0x7f2e001454a0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e001d7090 con 0x7f2e001454a0 2023-11-25T05:04:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e00145140 con 0x7f2e000a8e10 2023-11-25T05:04:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e001434b0 con 0x7f2e000a8a30 2023-11-25T05:04:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.412+0000 7f2e06a4b640 10 monclient(hunting): authenticate will time out at 1400.655577s 2023-11-25T05:04:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.413+0000 7f2dff7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e000a8e10 0x7f2e000ac9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.413+0000 7f2e04fe8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e001454a0 0x7f2e001d7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.413+0000 7f2dff7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e000a8e10 0x7f2e000ac9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:38266/0 (socket says 172.21.15.7:38266) 2023-11-25T05:04:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.413+0000 7f2dff7fe640 1 -- 172.21.15.7:0/2494450626 learned_addr learned my addr 172.21.15.7:0/2494450626 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.413+0000 7f2dff7fe640 10 monclient(hunting): get_auth_request con 0x7f2e000a8e10 auth_method 0 2023-11-25T05:04:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.413+0000 7f2dff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.413+0000 7f2dff7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.413+0000 7f2dff7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.413+0000 7f2e04fe8640 10 monclient(hunting): get_auth_request con 0x7f2e001454a0 auth_method 0 2023-11-25T05:04:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.413+0000 7f2e04fe8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.413+0000 7f2e04fe8640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.413+0000 7f2e04fe8640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.414+0000 7f2dff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.414+0000 7f2dff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.414+0000 7f2dff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.414+0000 7f2e04fe8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.414+0000 7f2e04fe8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.414+0000 7f2e04fe8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.416+0000 7f2dff7fe640 10 monclient(hunting): handle_auth_done global_id 4288 payload 824 2023-11-25T05:04:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.416+0000 7f2dff7fe640 1 -- 172.21.15.7:0/2494450626 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e000a8a30 msgr2=0x7f2e000abd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.416+0000 7f2dff7fe640 1 --2- 172.21.15.7:0/2494450626 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e000a8a30 0x7f2e000abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.416+0000 7f2dff7fe640 1 -- 172.21.15.7:0/2494450626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e001454a0 msgr2=0x7f2e001d7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.416+0000 7f2dff7fe640 1 --2- 172.21.15.7:0/2494450626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e001454a0 0x7f2e001d7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.416+0000 7f2dff7fe640 10 monclient: _finish_hunting 0 2023-11-25T05:04:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.416+0000 7f2dff7fe640 1 monclient: found mon.noname-b 2023-11-25T05:04:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.416+0000 7f2dff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.416+0000 7f2dff7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.416+0000 7f2dff7fe640 1 -- 172.21.15.7:0/2494450626 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df8009a70 con 0x7f2e000a8e10 2023-11-25T05:04:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.418+0000 7f2dff7fe640 1 --2- 172.21.15.7:0/2494450626 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e000a8e10 0x7f2e000ac9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2df400b4a0 tx=0x7f2df400b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.418+0000 7f2dfcff9640 10 client.?.objecter ms_handle_connect 0x7f2e000a8e10 2023-11-25T05:04:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.418+0000 7f2dfcff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2dfcff9640 1 -- 172.21.15.7:0/2494450626 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df4010610 con 0x7f2e000a8e10 2023-11-25T05:04:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2dfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2dfcff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2dfcff9640 10 monclient: dump: 2023-11-25T05:04:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:34.429 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2dfcff9640 10 monclient: _finish_auth 0 2023-11-25T05:04:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2dfcff9640 10 monclient: _check_auth_tickets 2023-11-25T05:04:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2dfcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2dfcff9640 1 -- 172.21.15.7:0/2494450626 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df4010770 con 0x7f2e000a8e10 2023-11-25T05:04:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2e06a4b640 5 monclient: authenticate success, global_id 4288 2023-11-25T05:04:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2dfcff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2dfcff9640 1 -- 172.21.15.7:0/2494450626 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df4010b20 con 0x7f2e000a8e10 2023-11-25T05:04:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2dfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2dfcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2dfcff9640 10 monclient: dump: 2023-11-25T05:04:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:34.431 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:34.431 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:34.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2e06a4b640 10 monclient: _renew_subs 2023-11-25T05:04:34.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2e06a4b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:34.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2e06a4b640 1 -- 172.21.15.7:0/2494450626 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e00143730 con 0x7f2e000a8e10 2023-11-25T05:04:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2dfe7fc640 4 set_mon_vals no callback set 2023-11-25T05:04:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2e06a4b640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2e06a4b640 10 monclient: _renew_subs 2023-11-25T05:04:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2e06a4b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2e06a4b640 1 -- 172.21.15.7:0/2494450626 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e000ad0e0 con 0x7f2e000a8e10 2023-11-25T05:04:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.422+0000 7f2e06a4b640 1 librados: init done 2023-11-25T05:04:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.423+0000 7f2e06a4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.423+0000 7f2e06a4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:34.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.423+0000 7f2e06a4b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:34.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.423+0000 7f2e06a4b640 1 -- 172.21.15.7:0/2494450626 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e00067ac0 con 0x7f2e000a8e10 2023-11-25T05:04:34.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.441+0000 7f2dfcff9640 1 -- 172.21.15.7:0/2494450626 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2df4010c80 con 0x7f2e000a8e10 2023-11-25T05:04:34.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.441+0000 7f2dfcff9640 1 --2- 172.21.15.7:0/2494450626 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f2dec06c850 0x7f2dec06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:34.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.442+0000 7f2dfffff640 1 --2- 172.21.15.7:0/2494450626 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f2dec06c850 0x7f2dec06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:34.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.442+0000 7f2dfffff640 10 monclient: get_auth_request con 0x7f2dec06c850 auth_method 0 2023-11-25T05:04:34.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.442+0000 7f2dfcff9640 10 client.4288.objecter ms_handle_connect 0x7f2dec06c850 2023-11-25T05:04:34.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.442+0000 7f2dfffff640 1 --2- 172.21.15.7:0/2494450626 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f2dec06c850 0x7f2dec06ed10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2df00094a0 tx=0x7f2df00093f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:34.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.446+0000 7f2dfcff9640 1 -- 172.21.15.7:0/2494450626 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2df4058380 con 0x7f2e000a8e10 2023-11-25T05:04:34.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.446+0000 7f2dfcff9640 10 client.4288.objecter ms_dispatch 0x7f2e00000b90 osd_map(21..21 src has 1..21) v4 2023-11-25T05:04:34.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.446+0000 7f2dfcff9640 3 client.4288.objecter handle_osd_map got epochs [21,21] > 0 2023-11-25T05:04:34.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.446+0000 7f2dfcff9640 3 client.4288.objecter handle_osd_map decoding full epoch 21 2023-11-25T05:04:34.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.446+0000 7f2dfcff9640 20 client.4288.objecter dump_active .. 0 homeless 2023-11-25T05:04:34.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.699+0000 7f2dfcff9640 1 -- 172.21.15.7:0/2494450626 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2df40546c0 con 0x7f2e000a8e10 2023-11-25T05:04:34.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.699+0000 7f2dfcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:34.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.699+0000 7f2dfcff9640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:34.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.877+0000 7f2e06a4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:34.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.877+0000 7f2e06a4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:34.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.877+0000 7f2e06a4b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:34.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.877+0000 7f2e06a4b640 1 -- 172.21.15.7:0/2494450626 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e001d6e60 con 0x7f2e000a8e10 2023-11-25T05:04:34.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.913+0000 7f2dfcff9640 1 -- 172.21.15.7:0/2494450626 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14543 (secure 0 0 0) 0x7f2e001d6e60 con 0x7f2e000a8e10 2023-11-25T05:04:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.913+0000 7f2dfcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.913+0000 7f2dfcff9640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:34.915 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:04:34.915 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"59cf1eff-816d-4eda-b513-a2de7ea218f7","created":"2023-11-25T05:03:35.598582+0000","modified":"2023-11-25T05:04:34.051490+0000","last_up_change":"2023-11-25T05:04:24.817743+0000","last_in_change":"2023-11-25T05:03:41.565581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-25T05:04:21.643061+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-25T05:04:31.232489+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5c87807e-7b82-4866-ba36-60b44a25f317","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6825","nonce":2271997805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6827","nonce":2271997805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6831","nonce":2271997805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6829","nonce":2271997805}]},"public_addr":"172.21.15.7:6825/2271997805","cluster_addr":"172.21.15.7:6827/2271997805","heartbeat_back_addr":"172.21.15.7:6831/2271997805","heartbeat_front_addr":"172.21.15.7:6829/2271997805","state":["exists","up"]},{"osd":1,"uuid":"0e252ab8-9524-4ccc-b08f-d95bbf05fbea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6801","nonce":3792760010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6803","nonce":3792760010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6807","nonce":3792760010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6805","nonce":3792760010}]},"public_addr":"172.21.15.7:6801/3792760010","cluster_addr":"172.21.15.7:6803/3792760010","heartbeat_back_addr":"172.21.15.7:6807/3792760010","heartbeat_front_addr":"172.21.15.7:6805/3792760010","state":["exists","up"]},{"osd":2,"uuid":"ccda96b8-aa9e-4c99-8d84-bba19ca0483b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6809","nonce":1372486177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6811","nonce":1372486177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6815","nonce":1372486177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6813","nonce":1372486177}]},"public_addr":"172.21.15.7:6809/1372486177","cluster_addr":"172.21.15.7:6811/1372486177","heartbeat_back_addr":"172.21.15.7:6815/1372486177","heartbeat_front_addr":"172.21.15.7:6813/1372486177","state":["exists","up"]},{"osd":3,"uuid":"e4b8700a-2e28-4e69-8577-465666b1b051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6817","nonce":59754388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6819","nonce":59754388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6823","nonce":59754388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6821","nonce":59754388}]},"public_addr":"172.21.15.7:6817/59754388","cluster_addr":"172.21.15.7:6819/59754388","heartbeat_back_addr":"172.21.15.7:6823/59754388","heartbeat_front_addr":"172.21.15.7:6821/59754388","state":["exists","up"]},{"osd":4,"uuid":"36c88a76-0467-4e37-85b4-dfd5953855a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6823","nonce":1576115380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6829","nonce":1576115380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6833","nonce":1576115380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6831","nonce":1576115380}]},"public_addr":"172.21.15.90:6823/1576115380","cluster_addr":"172.21.15.90:6829/1576115380","heartbeat_back_addr":"172.21.15.90:6833/1576115380","heartbeat_front_addr":"172.21.15.90:6831/1576115380","state":["exists","up"]},{"osd":5,"uuid":"ebf452a0-1b17-488a-bbea-2e953c932ac7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6811","nonce":132027349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6813","nonce":132027349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6817","nonce":132027349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6815","nonce":132027349}]},"public_addr":"172.21.15.90:6811/132027349","cluster_addr":"172.21.15.90:6813/132027349","heartbeat_back_addr":"172.21.15.90:6817/132027349","heartbeat_front_addr":"172.21.15.90:6815/132027349","state":["exists","up"]},{"osd":6,"uuid":"5aaf35c3-5c06-45e5-b788-8d3207c47cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6819","nonce":3560048676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6821","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6822","nonce":3560048676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6828","nonce":3560048676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6825","nonce":3560048676}]},"public_addr":"172.21.15.90:6819/3560048676","cluster_addr":"172.21.15.90:6822/3560048676","heartbeat_back_addr":"172.21.15.90:6828/3560048676","heartbeat_front_addr":"172.21.15.90:6825/3560048676","state":["exists","up"]},{"osd":7,"uuid":"d50dd23a-f864-4593-ac00-4d8afed21dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6803","nonce":815268397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6805","nonce":815268397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6809","nonce":815268397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6807","nonce":815268397}]},"public_addr":"172.21.15.90:6803/815268397","cluster_addr":"172.21.15.90:6805/815268397","heartbeat_back_addr":"172.21.15.90:6809/815268397","heartbeat_front_addr":"172.21.15.90:6807/815268397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:21.918591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:21.817203+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:21.018791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:22.357339+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:19.776736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:19.168168+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:20.275331+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:19.564073+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-25T05:04:34.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 10 librados: watch_flush enter 2023-11-25T05:04:34.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 10 librados: watch_flush exit 2023-11-25T05:04:34.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 20 client.4288.objecter shutdown clearing up homeless session... 2023-11-25T05:04:34.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 10 client.4288.objecter successfully canceled tick 2023-11-25T05:04:34.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 1 -- 172.21.15.7:0/2494450626 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f2dec06c850 msgr2=0x7f2dec06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:34.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 1 --2- 172.21.15.7:0/2494450626 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f2dec06c850 0x7f2dec06ed10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2df00094a0 tx=0x7f2df00093f0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 10 monclient: shutdown 2023-11-25T05:04:34.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:34.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 1 -- 172.21.15.7:0/2494450626 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e000a8e10 msgr2=0x7f2e000ac9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 1 --2- 172.21.15.7:0/2494450626 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e000a8e10 0x7f2e000ac9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2df400b4a0 tx=0x7f2df400b960 comp rx=0 tx=0).stop 2023-11-25T05:04:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 1 -- 172.21.15.7:0/2494450626 shutdown_connections 2023-11-25T05:04:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 1 --2- 172.21.15.7:0/2494450626 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f2dec06c850 0x7f2dec06ed10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 1 --2- 172.21.15.7:0/2494450626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e001454a0 0x7f2e001d7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 1 --2- 172.21.15.7:0/2494450626 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e000a8e10 0x7f2e000ac9c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 1 --2- 172.21.15.7:0/2494450626 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e000a8a30 0x7f2e000abd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 1 -- 172.21.15.7:0/2494450626 >> 172.21.15.7:0/2494450626 conn(0x7f2e00061a30 msgr2=0x7f2e0013cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 1 -- 172.21.15.7:0/2494450626 shutdown_connections 2023-11-25T05:04:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 1 -- 172.21.15.7:0/2494450626 wait complete. 2023-11-25T05:04:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:34.915+0000 7f2e06a4b640 1 librados: shutdown 2023-11-25T05:04:34.928 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-25T05:04:34.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-25T05:04:35.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.035+0000 7fe49c178640 10 monclient: get_monmap_and_config 2023-11-25T05:04:35.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.035+0000 7fe49c178640 10 monclient: build_initial_monmap 2023-11-25T05:04:35.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.035+0000 7fe49c178640 10 monclient: monmap: 2023-11-25T05:04:35.037 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:35.037 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.037 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:35.036489+0000 2023-11-25T05:04:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:35.036489+0000 2023-11-25T05:04:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:35.038 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.036+0000 7fe49c178640 1 Processor -- start 2023-11-25T05:04:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.036+0000 7fe49c178640 1 -- start start 2023-11-25T05:04:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.036+0000 7fe49c178640 10 monclient: init 2023-11-25T05:04:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.036+0000 7fe49c178640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.036+0000 7fe49c178640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-11-25T05:04:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.037+0000 7fe49c178640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4940a8910 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.037+0000 7fe49c178640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4940a9b70 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.037+0000 7fe49c178640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe49413f370 0x7fe4941426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.037+0000 7fe49c178640 10 monclient(hunting): picked mon.noname-c con 0x7fe49413f370 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.037+0000 7fe49c178640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4940a8cf0 con 0x7fe4940a8910 2023-11-25T05:04:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.037+0000 7fe49c178640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4940aabd0 con 0x7fe4940a9b70 2023-11-25T05:04:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.037+0000 7fe49c178640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.037+0000 7fe49c178640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4940aaeb0 con 0x7fe49413f370 2023-11-25T05:04:35.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.037+0000 7fe49c178640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:35.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.037+0000 7fe49c178640 10 monclient(hunting): authenticate will time out at 1401.279944s 2023-11-25T05:04:35.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.069+0000 7fe49a715640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe49413f370 0x7fe4941426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.069+0000 7fe499713640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe4940a9b70 0x7fe494141f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:52552/0 (socket says 172.21.15.7:52552) 2023-11-25T05:04:35.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.069+0000 7fe499713640 1 -- 172.21.15.7:0/3875004275 learned_addr learned my addr 172.21.15.7:0/3875004275 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:35.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.069+0000 7fe49a715640 10 monclient(hunting): get_auth_request con 0x7fe49413f370 auth_method 0 2023-11-25T05:04:35.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.069+0000 7fe49a715640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.069+0000 7fe49a715640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.069+0000 7fe49a715640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.070+0000 7fe49a715640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.070+0000 7fe49a715640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.070+0000 7fe49a715640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.072+0000 7fe49a715640 10 monclient(hunting): handle_auth_done global_id 4291 payload 274 2023-11-25T05:04:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.072+0000 7fe49a715640 1 -- 172.21.15.7:0/3875004275 >> v1:172.21.15.90:6789/0 conn(0x7fe4940a9b70 legacy=0x7fe494141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.072+0000 7fe49a715640 1 -- 172.21.15.7:0/3875004275 >> v1:172.21.15.7:6789/0 conn(0x7fe4940a8910 legacy=0x7fe49413ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.072+0000 7fe49a715640 10 monclient: _finish_hunting 0 2023-11-25T05:04:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.072+0000 7fe49a715640 1 monclient: found mon.noname-c 2023-11-25T05:04:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.072+0000 7fe49a715640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.072+0000 7fe49a715640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.072+0000 7fe49a715640 1 -- 172.21.15.7:0/3875004275 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4940a9720 con 0x7fe49413f370 2023-11-25T05:04:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.079+0000 7fe49a715640 1 --2- 172.21.15.7:0/3875004275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe49413f370 0x7fe4941426a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009d90 tx=0x7fe48c02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee7ea60bbf97a898 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe498f12640 1 -- 172.21.15.7:0/3875004275 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c02cd00 con 0x7fe49413f370 2023-11-25T05:04:35.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe498f12640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe498f12640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:35.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe498f12640 10 monclient: dump: 2023-11-25T05:04:35.098 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.098 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.098 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.098 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.098 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.098 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.098 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.098 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.099 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.099 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe498f12640 10 monclient: _finish_auth 0 2023-11-25T05:04:35.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe498f12640 10 monclient: _check_auth_tickets 2023-11-25T05:04:35.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe498f12640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:35.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe498f12640 1 -- 172.21.15.7:0/3875004275 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48c02ce60 con 0x7fe49413f370 2023-11-25T05:04:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe49c178640 5 monclient: authenticate success, global_id 4291 2023-11-25T05:04:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe49c178640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe498f12640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe49c178640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe49c178640 4 set_mon_vals no callback set 2023-11-25T05:04:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe498f12640 1 -- 172.21.15.7:0/3875004275 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c034660 con 0x7fe49413f370 2023-11-25T05:04:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe498f12640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe498f12640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.085+0000 7fe498f12640 10 monclient: dump: 2023-11-25T05:04:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.101 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.086+0000 7fe49c178640 10 monclient: shutdown 2023-11-25T05:04:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.086+0000 7fe49c178640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.086+0000 7fe49c178640 1 -- 172.21.15.7:0/3875004275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe49413f370 msgr2=0x7fe4941426a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.086+0000 7fe49c178640 1 --2- 172.21.15.7:0/3875004275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe49413f370 0x7fe4941426a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009d90 tx=0x7fe48c02bf50 comp rx=0 tx=0).stop 2023-11-25T05:04:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.086+0000 7fe49c178640 1 -- 172.21.15.7:0/3875004275 shutdown_connections 2023-11-25T05:04:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.086+0000 7fe49c178640 1 --2- 172.21.15.7:0/3875004275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe49413f370 0x7fe4941426a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.086+0000 7fe49c178640 1 -- 172.21.15.7:0/3875004275 >> 172.21.15.7:0/3875004275 conn(0x7fe4940b7780 msgr2=0x7fe4940b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.086+0000 7fe49c178640 1 -- 172.21.15.7:0/3875004275 shutdown_connections 2023-11-25T05:04:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.086+0000 7fe49c178640 1 -- 172.21.15.7:0/3875004275 wait complete. 2023-11-25T05:04:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.087+0000 7fe49c178640 10 monclient: build_initial_monmap 2023-11-25T05:04:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.087+0000 7fe49c178640 10 monclient: monmap: 2023-11-25T05:04:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:35.102 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.087+0000 7fe49c178640 1 librados: starting msgr at 2023-11-25T05:04:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.087+0000 7fe49c178640 1 librados: starting objecter 2023-11-25T05:04:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.087+0000 7fe49c178640 1 Processor -- start 2023-11-25T05:04:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.087+0000 7fe49c178640 1 -- start start 2023-11-25T05:04:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.087+0000 7fe49c178640 1 librados: setting wanted keys 2023-11-25T05:04:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.087+0000 7fe49c178640 1 librados: calling monclient init 2023-11-25T05:04:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.087+0000 7fe49c178640 10 monclient: init 2023-11-25T05:04:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-25T05:04:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4940a8910 0x7fe49413a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 10 monclient(hunting): picked mon.noname-a con 0x7fe4940a8910 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe49413f370 0x7fe49413ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 10 monclient(hunting): picked mon.noname-b con 0x7fe49413f370 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe4940b1430 0x7fe4940ac3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 10 monclient(hunting): picked mon.noname-c con 0x7fe4940b1430 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4941391c0 con 0x7fe4940a8910 2023-11-25T05:04:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49413b160 con 0x7fe49413f370 2023-11-25T05:04:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4941396d0 con 0x7fe4940b1430 2023-11-25T05:04:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49c178640 10 monclient(hunting): authenticate will time out at 1401.331180s 2023-11-25T05:04:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.088+0000 7fe49a715640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe4940b1430 0x7fe4940ac3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.089+0000 7fe49a715640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe4940b1430 0x7fe4940ac3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33572/0 (socket says 172.21.15.7:33572) 2023-11-25T05:04:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.089+0000 7fe49a715640 1 -- 172.21.15.7:0/921847910 learned_addr learned my addr 172.21.15.7:0/921847910 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:35.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.089+0000 7fe49a715640 10 monclient(hunting): get_auth_request con 0x7fe4940b1430 auth_method 0 2023-11-25T05:04:35.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.089+0000 7fe49a715640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.089+0000 7fe49a715640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.089+0000 7fe49a715640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.089+0000 7fe49a715640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.089+0000 7fe49a715640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.089+0000 7fe49a715640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.089+0000 7fe499713640 1 --2- 172.21.15.7:0/921847910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe49413f370 0x7fe49413ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.090+0000 7fe499713640 10 monclient(hunting): get_auth_request con 0x7fe49413f370 auth_method 0 2023-11-25T05:04:35.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.090+0000 7fe499713640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.090+0000 7fe499713640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.090+0000 7fe499713640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.090+0000 7fe499f14640 1 --2- 172.21.15.7:0/921847910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4940a8910 0x7fe49413a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.090+0000 7fe499713640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.090+0000 7fe499713640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.090+0000 7fe499713640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.090+0000 7fe499f14640 10 monclient(hunting): get_auth_request con 0x7fe4940a8910 auth_method 0 2023-11-25T05:04:35.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.090+0000 7fe499f14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.090+0000 7fe499f14640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.090+0000 7fe499f14640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.091+0000 7fe499f14640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.091+0000 7fe499f14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.091+0000 7fe499f14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.092+0000 7fe49a715640 10 monclient(hunting): handle_auth_done global_id 4244 payload 824 2023-11-25T05:04:35.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.092+0000 7fe49a715640 1 -- 172.21.15.7:0/921847910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe49413f370 msgr2=0x7fe49413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.092+0000 7fe49a715640 1 --2- 172.21.15.7:0/921847910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe49413f370 0x7fe49413ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.092+0000 7fe49a715640 1 -- 172.21.15.7:0/921847910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4940a8910 msgr2=0x7fe49413a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.092+0000 7fe49a715640 1 --2- 172.21.15.7:0/921847910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4940a8910 0x7fe49413a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.092+0000 7fe49a715640 10 monclient: _finish_hunting 0 2023-11-25T05:04:35.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.092+0000 7fe49a715640 1 monclient: found mon.noname-c 2023-11-25T05:04:35.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.092+0000 7fe49a715640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:35.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.092+0000 7fe49a715640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:04:35.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.092+0000 7fe49a715640 1 -- 172.21.15.7:0/921847910 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c009a70 con 0x7fe4940b1430 2023-11-25T05:04:35.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.095+0000 7fe49a715640 1 --2- 172.21.15.7:0/921847910 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe4940b1430 0x7fe4940ac3b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009eb0 tx=0x7fe48c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.095+0000 7fe48a7fc640 10 client.?.objecter ms_handle_connect 0x7fe4940b1430 2023-11-25T05:04:35.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.095+0000 7fe48a7fc640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:35.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe48a7fc640 1 -- 172.21.15.7:0/921847910 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c03d600 con 0x7fe4940b1430 2023-11-25T05:04:35.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe48a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe48a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:35.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe48a7fc640 10 monclient: dump: 2023-11-25T05:04:35.111 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.111 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.111 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.111 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.112 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.112 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.112 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.112 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.112 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.112 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe48a7fc640 10 monclient: _finish_auth 0 2023-11-25T05:04:35.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe48a7fc640 10 monclient: _check_auth_tickets 2023-11-25T05:04:35.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe48a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:35.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe48a7fc640 1 -- 172.21.15.7:0/921847910 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48c03d780 con 0x7fe4940b1430 2023-11-25T05:04:35.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe49c178640 5 monclient: authenticate success, global_id 4244 2023-11-25T05:04:35.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe49c178640 10 monclient: _renew_subs 2023-11-25T05:04:35.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe49c178640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:35.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe49c178640 1 -- 172.21.15.7:0/921847910 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4940ac8f0 con 0x7fe4940b1430 2023-11-25T05:04:35.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe48a7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:35.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe48a7fc640 1 -- 172.21.15.7:0/921847910 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c03db30 con 0x7fe4940b1430 2023-11-25T05:04:35.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe48a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe48a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:35.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe48a7fc640 10 monclient: dump: 2023-11-25T05:04:35.113 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.113 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.114 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.114 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.114 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.114 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.114 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.114 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.114 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.114 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe48bfff640 4 set_mon_vals no callback set 2023-11-25T05:04:35.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe49c178640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:35.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe49c178640 10 monclient: _renew_subs 2023-11-25T05:04:35.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe49c178640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:35.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe49c178640 1 -- 172.21.15.7:0/921847910 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4941dab50 con 0x7fe4940b1430 2023-11-25T05:04:35.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.099+0000 7fe49c178640 1 librados: init done 2023-11-25T05:04:35.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.100+0000 7fe49c178640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:35.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.100+0000 7fe49c178640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:35.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.100+0000 7fe49c178640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:35.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.100+0000 7fe49c178640 1 -- 172.21.15.7:0/921847910 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe494139330 con 0x7fe4940b1430 2023-11-25T05:04:35.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.118+0000 7fe48a7fc640 1 -- 172.21.15.7:0/921847910 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe48c03b040 con 0x7fe4940b1430 2023-11-25T05:04:35.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.119+0000 7fe48a7fc640 1 --2- 172.21.15.7:0/921847910 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe47806c820 0x7fe47806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.119+0000 7fe499f14640 1 --2- 172.21.15.7:0/921847910 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe47806c820 0x7fe47806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.119+0000 7fe499f14640 10 monclient: get_auth_request con 0x7fe47806c820 auth_method 0 2023-11-25T05:04:35.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.119+0000 7fe499f14640 1 --2- 172.21.15.7:0/921847910 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe47806c820 0x7fe47806ece0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe490000f60 tx=0x7fe490009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.119+0000 7fe48a7fc640 10 client.4244.objecter ms_handle_connect 0x7fe47806c820 2023-11-25T05:04:35.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.121+0000 7fe48a7fc640 1 -- 172.21.15.7:0/921847910 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe48c07aea0 con 0x7fe4940b1430 2023-11-25T05:04:35.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.121+0000 7fe48a7fc640 10 client.4244.objecter ms_dispatch 0x7fe494000b90 osd_map(21..21 src has 1..21) v4 2023-11-25T05:04:35.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.121+0000 7fe48a7fc640 3 client.4244.objecter handle_osd_map got epochs [21,21] > 0 2023-11-25T05:04:35.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.121+0000 7fe48a7fc640 3 client.4244.objecter handle_osd_map decoding full epoch 21 2023-11-25T05:04:35.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.121+0000 7fe48a7fc640 20 client.4244.objecter dump_active .. 0 homeless 2023-11-25T05:04:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.387+0000 7fe48a7fc640 1 -- 172.21.15.7:0/921847910 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe48c0771e0 con 0x7fe4940b1430 2023-11-25T05:04:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.387+0000 7fe48a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.387+0000 7fe48a7fc640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:35.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.591+0000 7fe49c178640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:35.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.591+0000 7fe49c178640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:35.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.591+0000 7fe49c178640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:35.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.591+0000 7fe49c178640 1 -- 172.21.15.7:0/921847910 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe494075b60 con 0x7fe4940b1430 2023-11-25T05:04:35.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.634+0000 7fe48a7fc640 1 -- 172.21.15.7:0/921847910 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14543 (secure 0 0 0) 0x7fe48c04e090 con 0x7fe4940b1430 2023-11-25T05:04:35.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.634+0000 7fe48a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:04:35.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.634+0000 7fe48a7fc640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:35.637 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:04:35.637 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"59cf1eff-816d-4eda-b513-a2de7ea218f7","created":"2023-11-25T05:03:35.598582+0000","modified":"2023-11-25T05:04:34.051490+0000","last_up_change":"2023-11-25T05:04:24.817743+0000","last_in_change":"2023-11-25T05:03:41.565581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-25T05:04:21.643061+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-25T05:04:31.232489+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5c87807e-7b82-4866-ba36-60b44a25f317","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6825","nonce":2271997805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6827","nonce":2271997805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6831","nonce":2271997805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6829","nonce":2271997805}]},"public_addr":"172.21.15.7:6825/2271997805","cluster_addr":"172.21.15.7:6827/2271997805","heartbeat_back_addr":"172.21.15.7:6831/2271997805","heartbeat_front_addr":"172.21.15.7:6829/2271997805","state":["exists","up"]},{"osd":1,"uuid":"0e252ab8-9524-4ccc-b08f-d95bbf05fbea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6801","nonce":3792760010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6803","nonce":3792760010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6807","nonce":3792760010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6805","nonce":3792760010}]},"public_addr":"172.21.15.7:6801/3792760010","cluster_addr":"172.21.15.7:6803/3792760010","heartbeat_back_addr":"172.21.15.7:6807/3792760010","heartbeat_front_addr":"172.21.15.7:6805/3792760010","state":["exists","up"]},{"osd":2,"uuid":"ccda96b8-aa9e-4c99-8d84-bba19ca0483b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6809","nonce":1372486177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6811","nonce":1372486177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6815","nonce":1372486177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6813","nonce":1372486177}]},"public_addr":"172.21.15.7:6809/1372486177","cluster_addr":"172.21.15.7:6811/1372486177","heartbeat_back_addr":"172.21.15.7:6815/1372486177","heartbeat_front_addr":"172.21.15.7:6813/1372486177","state":["exists","up"]},{"osd":3,"uuid":"e4b8700a-2e28-4e69-8577-465666b1b051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6817","nonce":59754388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6819","nonce":59754388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6823","nonce":59754388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6821","nonce":59754388}]},"public_addr":"172.21.15.7:6817/59754388","cluster_addr":"172.21.15.7:6819/59754388","heartbeat_back_addr":"172.21.15.7:6823/59754388","heartbeat_front_addr":"172.21.15.7:6821/59754388","state":["exists","up"]},{"osd":4,"uuid":"36c88a76-0467-4e37-85b4-dfd5953855a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6823","nonce":1576115380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6829","nonce":1576115380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6833","nonce":1576115380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6831","nonce":1576115380}]},"public_addr":"172.21.15.90:6823/1576115380","cluster_addr":"172.21.15.90:6829/1576115380","heartbeat_back_addr":"172.21.15.90:6833/1576115380","heartbeat_front_addr":"172.21.15.90:6831/1576115380","state":["exists","up"]},{"osd":5,"uuid":"ebf452a0-1b17-488a-bbea-2e953c932ac7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6811","nonce":132027349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6813","nonce":132027349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6817","nonce":132027349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6815","nonce":132027349}]},"public_addr":"172.21.15.90:6811/132027349","cluster_addr":"172.21.15.90:6813/132027349","heartbeat_back_addr":"172.21.15.90:6817/132027349","heartbeat_front_addr":"172.21.15.90:6815/132027349","state":["exists","up"]},{"osd":6,"uuid":"5aaf35c3-5c06-45e5-b788-8d3207c47cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6819","nonce":3560048676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6821","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6822","nonce":3560048676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6828","nonce":3560048676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6825","nonce":3560048676}]},"public_addr":"172.21.15.90:6819/3560048676","cluster_addr":"172.21.15.90:6822/3560048676","heartbeat_back_addr":"172.21.15.90:6828/3560048676","heartbeat_front_addr":"172.21.15.90:6825/3560048676","state":["exists","up"]},{"osd":7,"uuid":"d50dd23a-f864-4593-ac00-4d8afed21dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6803","nonce":815268397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6805","nonce":815268397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6809","nonce":815268397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6807","nonce":815268397}]},"public_addr":"172.21.15.90:6803/815268397","cluster_addr":"172.21.15.90:6805/815268397","heartbeat_back_addr":"172.21.15.90:6809/815268397","heartbeat_front_addr":"172.21.15.90:6807/815268397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:21.918591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:21.817203+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:21.018791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:22.357339+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:19.776736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:19.168168+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:20.275331+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:19.564073+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-25T05:04:35.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.636+0000 7fe49c178640 10 librados: watch_flush enter 2023-11-25T05:04:35.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.636+0000 7fe49c178640 10 librados: watch_flush exit 2023-11-25T05:04:35.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.636+0000 7fe49c178640 20 client.4244.objecter shutdown clearing up homeless session... 2023-11-25T05:04:35.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.636+0000 7fe49c178640 10 client.4244.objecter successfully canceled tick 2023-11-25T05:04:35.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.636+0000 7fe49c178640 1 -- 172.21.15.7:0/921847910 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe47806c820 msgr2=0x7fe47806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.636+0000 7fe49c178640 1 --2- 172.21.15.7:0/921847910 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe47806c820 0x7fe47806ece0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe490000f60 tx=0x7fe490009040 comp rx=0 tx=0).stop 2023-11-25T05:04:35.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.637+0000 7fe49c178640 10 monclient: shutdown 2023-11-25T05:04:35.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.637+0000 7fe49c178640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:35.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.637+0000 7fe49c178640 1 -- 172.21.15.7:0/921847910 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe4940b1430 msgr2=0x7fe4940ac3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.637+0000 7fe49c178640 1 --2- 172.21.15.7:0/921847910 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe4940b1430 0x7fe4940ac3b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009eb0 tx=0x7fe48c004a40 comp rx=0 tx=0).stop 2023-11-25T05:04:35.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.637+0000 7fe49c178640 1 -- 172.21.15.7:0/921847910 shutdown_connections 2023-11-25T05:04:35.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.637+0000 7fe49c178640 1 --2- 172.21.15.7:0/921847910 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe47806c820 0x7fe47806ece0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.637+0000 7fe49c178640 1 --2- 172.21.15.7:0/921847910 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe4940b1430 0x7fe4940ac3b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.637+0000 7fe49c178640 1 --2- 172.21.15.7:0/921847910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe49413f370 0x7fe49413ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.637+0000 7fe49c178640 1 --2- 172.21.15.7:0/921847910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4940a8910 0x7fe49413a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.637+0000 7fe49c178640 1 -- 172.21.15.7:0/921847910 >> 172.21.15.7:0/921847910 conn(0x7fe4940b7780 msgr2=0x7fe4941373b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:35.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.637+0000 7fe49c178640 1 -- 172.21.15.7:0/921847910 shutdown_connections 2023-11-25T05:04:35.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.637+0000 7fe49c178640 1 -- 172.21.15.7:0/921847910 wait complete. 2023-11-25T05:04:35.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.637+0000 7fe49c178640 1 librados: shutdown 2023-11-25T05:04:35.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-25T05:04:35.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-25T05:04:35.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-25T05:04:35.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-11-25T05:04:35.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-11-25T05:04:35.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-11-25T05:04:35.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-11-25T05:04:35.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-11-25T05:04:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.856+0000 7f9dda6c2640 10 monclient: get_monmap_and_config 2023-11-25T05:04:35.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.856+0000 7f9dda6c2640 10 monclient: build_initial_monmap 2023-11-25T05:04:35.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.856+0000 7f9dda6c2640 10 monclient: monmap: 2023-11-25T05:04:35.864 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:35.865 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.866 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:35.857165+0000 2023-11-25T05:04:35.866 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:35.857165+0000 2023-11-25T05:04:35.866 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:35.866 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.866 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:35.866 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:35.866 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:35.866 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:35.866 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:35.867 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.857+0000 7f465694e640 10 monclient: get_monmap_and_config 2023-11-25T05:04:35.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.857+0000 7f465694e640 10 monclient: build_initial_monmap 2023-11-25T05:04:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.857+0000 7f465694e640 10 monclient: monmap: 2023-11-25T05:04:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:35.858863+0000 2023-11-25T05:04:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:35.858863+0000 2023-11-25T05:04:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:35.868 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.859+0000 7f465694e640 1 Processor -- start 2023-11-25T05:04:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.859+0000 7f465694e640 1 -- start start 2023-11-25T05:04:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.859+0000 7f465694e640 10 monclient: init 2023-11-25T05:04:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.860+0000 7f465694e640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.860+0000 7f465694e640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-11-25T05:04:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.860+0000 7f465694e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46500af360 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.860+0000 7f465694e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f465006f2b0 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.860+0000 7f465694e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f465014a360 0x7f4650151a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.860+0000 7f465694e640 10 monclient(hunting): picked mon.noname-a con 0x7f465014a360 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.860+0000 7f465694e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4650143e60 con 0x7f465006f2b0 2023-11-25T05:04:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.860+0000 7f465694e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4650070fc0 con 0x7f46500af360 2023-11-25T05:04:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.860+0000 7f465694e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.860+0000 7f465694e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46500af740 con 0x7f465014a360 2023-11-25T05:04:35.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.860+0000 7f465694e640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:35.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.860+0000 7f465694e640 10 monclient(hunting): authenticate will time out at 1402.103230s 2023-11-25T05:04:35.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f464f7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f465006f2b0 0x7f4650149c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37508/0 (socket says 172.21.15.7:37508) 2023-11-25T05:04:35.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f464f7fe640 1 -- 172.21.15.7:0/64761080 learned_addr learned my addr 172.21.15.7:0/64761080 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:35.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f4654eeb640 1 --2- 172.21.15.7:0/64761080 >> v2:172.21.15.7:3300/0 conn(0x7f465014a360 0x7f4650151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.861+0000 7f9dda6c2640 1 Processor -- start 2023-11-25T05:04:35.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.861+0000 7f9dda6c2640 1 -- start start 2023-11-25T05:04:35.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.861+0000 7f9dda6c2640 10 monclient: init 2023-11-25T05:04:35.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f9dda6c2640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:35.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f9dda6c2640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2023-11-25T05:04:35.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f4654eeb640 10 monclient(hunting): get_auth_request con 0x7f465014a360 auth_method 0 2023-11-25T05:04:35.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f4654eeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f4654eeb640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f4654eeb640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f9dda6c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9dd40af360 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:35.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f9dda6c2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9dd4143920 0x7f9dd4149880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f9dda6c2640 10 monclient(hunting): picked mon.noname-a con 0x7f9dd4143920 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:35.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f9dda6c2640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9dd4149dc0 0x7f9dd41509f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f9dda6c2640 10 monclient(hunting): picked mon.noname-c con 0x7f9dd4149dc0 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:35.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f9dda6c2640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd4143d40 con 0x7f9dd40af360 2023-11-25T05:04:35.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f9dda6c2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f9dda6c2640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd40af740 con 0x7f9dd4143920 2023-11-25T05:04:35.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f9dda6c2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f9dda6c2640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd4070c50 con 0x7f9dd4149dc0 2023-11-25T05:04:35.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f9dda6c2640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:35.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.862+0000 7f9dda6c2640 10 monclient(hunting): authenticate will time out at 1402.105206s 2023-11-25T05:04:35.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.863+0000 7f9dd3fff640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9dd40af360 0x7f9dd4143210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:52568/0 (socket says 172.21.15.7:52568) 2023-11-25T05:04:35.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.863+0000 7f4654eeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.863+0000 7f4654eeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.863+0000 7f4654eeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.863+0000 7f9dd3fff640 1 -- 172.21.15.7:0/2515178051 learned_addr learned my addr 172.21.15.7:0/2515178051 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:35.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.863+0000 7f9dd8c5f640 1 --2- 172.21.15.7:0/2515178051 >> v2:172.21.15.90:3300/0 conn(0x7f9dd4149dc0 0x7f9dd41509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.864+0000 7feb98426640 10 monclient: get_monmap_and_config 2023-11-25T05:04:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.864+0000 7feb98426640 10 monclient: build_initial_monmap 2023-11-25T05:04:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.864+0000 7feb98426640 10 monclient: monmap: 2023-11-25T05:04:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:35.865203+0000 2023-11-25T05:04:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:35.865203+0000 2023-11-25T05:04:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:35.877 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:35.877 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:35.877 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.865+0000 7f9dd8c5f640 10 monclient(hunting): get_auth_request con 0x7f9dd4149dc0 auth_method 0 2023-11-25T05:04:35.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.865+0000 7f9dd8c5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.865+0000 7f9dd8c5f640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.865+0000 7f9dd8c5f640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.865+0000 7feb98426640 1 Processor -- start 2023-11-25T05:04:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.865+0000 7feb98426640 1 -- start start 2023-11-25T05:04:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.865+0000 7feb98426640 10 monclient: init 2023-11-25T05:04:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7f4654eeb640 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2023-11-25T05:04:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7f4654eeb640 1 -- 172.21.15.7:0/64761080 >> v1:172.21.15.90:6789/0 conn(0x7f46500af360 legacy=0x7f46501469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7feb98426640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7feb98426640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-11-25T05:04:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7feb98426640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb900af700 0x7feb90141670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7feb98426640 10 monclient(hunting): picked mon.noname-b con 0x7feb900af700 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7feb98426640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb90070220 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7feb98426640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7feb90072330 0x7feb90148a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7feb98426640 10 monclient(hunting): picked mon.noname-c con 0x7feb90072330 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7feb98426640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb900717e0 con 0x7feb90070220 2023-11-25T05:04:35.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7feb98426640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7feb98426640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb9006f950 con 0x7feb90072330 2023-11-25T05:04:35.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7feb98426640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7feb98426640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb900afae0 con 0x7feb900af700 2023-11-25T05:04:35.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7feb98426640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:35.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7feb98426640 10 monclient(hunting): authenticate will time out at 1402.109538s 2023-11-25T05:04:35.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7f4654eeb640 1 -- 172.21.15.7:0/64761080 >> v1:172.21.15.7:6789/0 conn(0x7f465006f2b0 legacy=0x7f4650149c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:35.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7f4654eeb640 10 monclient: _finish_hunting 0 2023-11-25T05:04:35.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7f4654eeb640 1 monclient: found mon.noname-a 2023-11-25T05:04:35.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7f4654eeb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:35.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7f4654eeb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:35.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.866+0000 7f4654eeb640 1 -- 172.21.15.7:0/64761080 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4650149730 con 0x7f465014a360 2023-11-25T05:04:35.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.867+0000 7f9dd8c5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.867+0000 7f9dd8c5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.867+0000 7f9dd8c5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.867+0000 7feb961c2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb900af700 0x7feb90141670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.867+0000 7f9dd37fe640 1 --2- 172.21.15.7:0/2515178051 >> v2:172.21.15.7:3300/0 conn(0x7f9dd4143920 0x7f9dd4149880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.868+0000 7feb961c2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb900af700 0x7feb90141670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33574/0 (socket says 172.21.15.7:33574) 2023-11-25T05:04:35.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.868+0000 7feb961c2640 1 -- 172.21.15.7:0/1186408509 learned_addr learned my addr 172.21.15.7:0/1186408509 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:35.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.868+0000 7feb961c2640 10 monclient(hunting): get_auth_request con 0x7feb900af700 auth_method 0 2023-11-25T05:04:35.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.868+0000 7feb961c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.868+0000 7feb961c2640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.868+0000 7feb961c2640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.868+0000 7f9dd37fe640 10 monclient(hunting): get_auth_request con 0x7f9dd4143920 auth_method 0 2023-11-25T05:04:35.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.868+0000 7f9dd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.868+0000 7f9dd37fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.868+0000 7f9dd37fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.869+0000 7feb969c3640 1 --2- 172.21.15.7:0/1186408509 >> v2:172.21.15.90:3300/0 conn(0x7feb90072330 0x7feb90148a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.869+0000 7feb961c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.869+0000 7feb961c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.869+0000 7feb961c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.869+0000 7f9dd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.869+0000 7f9dd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.869+0000 7f9dd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.869+0000 7f9dd8c5f640 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2023-11-25T05:04:35.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.869+0000 7f9dd8c5f640 1 -- 172.21.15.7:0/2515178051 >> v2:172.21.15.7:3300/0 conn(0x7f9dd4143920 msgr2=0x7f9dd4149880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:35.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.869+0000 7f9dd8c5f640 1 --2- 172.21.15.7:0/2515178051 >> v2:172.21.15.7:3300/0 conn(0x7f9dd4143920 0x7f9dd4149880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.869+0000 7f9dd8c5f640 1 -- 172.21.15.7:0/2515178051 >> v1:172.21.15.90:6789/0 conn(0x7f9dd40af360 legacy=0x7f9dd4143210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.870+0000 7f9dd8c5f640 10 monclient: _finish_hunting 0 2023-11-25T05:04:35.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.870+0000 7f9dd8c5f640 1 monclient: found mon.noname-c 2023-11-25T05:04:35.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.870+0000 7f9dd8c5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:35.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.870+0000 7f9dd8c5f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:35.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.870+0000 7f9dd8c5f640 1 -- 172.21.15.7:0/2515178051 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd4070f30 con 0x7f9dd4149dc0 2023-11-25T05:04:35.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.871+0000 7feb969c3640 10 monclient(hunting): get_auth_request con 0x7feb90072330 auth_method 0 2023-11-25T05:04:35.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.871+0000 7feb969c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.871+0000 7feb969c3640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.871+0000 7feb969c3640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.871+0000 7feb961c2640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2023-11-25T05:04:35.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.871+0000 7feb961c2640 1 -- 172.21.15.7:0/1186408509 >> v2:172.21.15.90:3300/0 conn(0x7feb90072330 msgr2=0x7feb90148a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:35.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.871+0000 7feb961c2640 1 --2- 172.21.15.7:0/1186408509 >> v2:172.21.15.90:3300/0 conn(0x7feb90072330 0x7feb90148a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.871+0000 7feb961c2640 1 -- 172.21.15.7:0/1186408509 >> v1:172.21.15.7:6789/0 conn(0x7feb90070220 legacy=0x7feb9013b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:35.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.871+0000 7feb961c2640 10 monclient: _finish_hunting 0 2023-11-25T05:04:35.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.871+0000 7feb961c2640 1 monclient: found mon.noname-b 2023-11-25T05:04:35.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.871+0000 7feb961c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:35.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.871+0000 7feb961c2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:04:35.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.871+0000 7feb961c2640 1 -- 172.21.15.7:0/1186408509 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb90071e50 con 0x7feb900af700 2023-11-25T05:04:35.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.874+0000 7feb961c2640 1 --2- 172.21.15.7:0/1186408509 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb900af700 0x7feb90141670 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0098a0 tx=0x7feb8c02b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c673876ffbb6d26d server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.874+0000 7f9dd8c5f640 1 --2- 172.21.15.7:0/2515178051 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd4149dc0 0x7f9dd41509f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9dd41460e0 tx=0x7f9dc802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aae0b24be0332a70 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.877+0000 7feb951c0640 1 -- 172.21.15.7:0/1186408509 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8c0317e0 con 0x7feb900af700 2023-11-25T05:04:35.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.877+0000 7feb951c0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.877+0000 7feb951c0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:35.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.877+0000 7feb951c0640 10 monclient: dump: 2023-11-25T05:04:35.889 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.889 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.889 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.889 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.889 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.889 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.889 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.889 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.889 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.890 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.877+0000 7feb951c0640 10 monclient: _finish_auth 0 2023-11-25T05:04:35.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.877+0000 7feb951c0640 10 monclient: _check_auth_tickets 2023-11-25T05:04:35.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.877+0000 7feb951c0640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:35.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.877+0000 7feb98426640 5 monclient: authenticate success, global_id 4247 2023-11-25T05:04:35.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.877+0000 7feb98426640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.878+0000 7fa22a36a640 10 monclient: get_monmap_and_config 2023-11-25T05:04:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.878+0000 7fa22a36a640 10 monclient: build_initial_monmap 2023-11-25T05:04:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.878+0000 7fa22a36a640 10 monclient: monmap: 2023-11-25T05:04:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:35.879387+0000 2023-11-25T05:04:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:35.879387+0000 2023-11-25T05:04:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:35.892 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:35.892 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:35.892 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:35.892 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:35.892 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.879+0000 7fa22a36a640 1 Processor -- start 2023-11-25T05:04:35.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.879+0000 7fa22a36a640 1 -- start start 2023-11-25T05:04:35.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.879+0000 7fa22a36a640 10 monclient: init 2023-11-25T05:04:35.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.879+0000 7f4654eeb640 1 --2- 172.21.15.7:0/64761080 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f465014a360 0x7f4650151a70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f46440097a0 tx=0x7f464402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e0f3e18dd311b7d server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7fa22a36a640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:35.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7fa22a36a640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-11-25T05:04:35.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7fa22a36a640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa2240a8830 0x7fa2241423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7fa22a36a640 10 monclient(hunting): picked mon.noname-c con 0x7fa2240a8830 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:35.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7fa22a36a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa2240b2110 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:35.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7fa22a36a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa2240a8c10 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:35.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7fa22a36a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2240aab10 con 0x7fa2240a8c10 2023-11-25T05:04:35.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7fa22a36a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa224141f00 con 0x7fa2240b2110 2023-11-25T05:04:35.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7fa22a36a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7fa22a36a640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2240aad40 con 0x7fa2240a8830 2023-11-25T05:04:35.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7fa22a36a640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:35.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7fa22a36a640 10 monclient(hunting): authenticate will time out at 1402.123317s 2023-11-25T05:04:35.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7feb951c0640 1 -- 172.21.15.7:0/1186408509 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8c031940 con 0x7feb900af700 2023-11-25T05:04:35.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7feb951c0640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:35.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7feb98426640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:35.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7feb98426640 4 set_mon_vals no callback set 2023-11-25T05:04:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7feb951c0640 1 -- 172.21.15.7:0/1186408509 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8c031cd0 con 0x7feb900af700 2023-11-25T05:04:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7feb951c0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7feb951c0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.880+0000 7feb951c0640 10 monclient: dump: 2023-11-25T05:04:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.896 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.881+0000 7fa229368640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa2240a8830 0x7fa2241423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.881+0000 7fa228b67640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa2240b2110 0x7fa22413ef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:52574/0 (socket says 172.21.15.7:52574) 2023-11-25T05:04:35.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.881+0000 7fa228b67640 1 -- 172.21.15.7:0/928535431 learned_addr learned my addr 172.21.15.7:0/928535431 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:35.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.881+0000 7feb98426640 10 monclient: shutdown 2023-11-25T05:04:35.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.881+0000 7feb98426640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:35.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.881+0000 7feb98426640 1 -- 172.21.15.7:0/1186408509 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb900af700 msgr2=0x7feb90141670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.881+0000 7feb98426640 1 --2- 172.21.15.7:0/1186408509 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb900af700 0x7feb90141670 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0098a0 tx=0x7feb8c02b880 comp rx=0 tx=0).stop 2023-11-25T05:04:35.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.881+0000 7fa229368640 10 monclient(hunting): get_auth_request con 0x7fa2240a8830 auth_method 0 2023-11-25T05:04:35.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.881+0000 7fa229368640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.881+0000 7fa229368640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.881+0000 7fa229368640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.882+0000 7feb98426640 1 -- 172.21.15.7:0/1186408509 shutdown_connections 2023-11-25T05:04:35.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.882+0000 7feb98426640 1 --2- 172.21.15.7:0/1186408509 >> v2:172.21.15.90:3300/0 conn(0x7feb90072330 0x7feb90148a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.882+0000 7feb98426640 1 --2- 172.21.15.7:0/1186408509 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb900af700 0x7feb90141670 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.882+0000 7feb98426640 1 -- 172.21.15.7:0/1186408509 >> 172.21.15.7:0/1186408509 conn(0x7feb9006a6e0 msgr2=0x7feb9006aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:35.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.882+0000 7feb98426640 1 -- 172.21.15.7:0/1186408509 shutdown_connections 2023-11-25T05:04:35.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.883+0000 7fa229368640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.883+0000 7fa229368640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.883+0000 7fa229368640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.882+0000 7feb98426640 1 -- 172.21.15.7:0/1186408509 wait complete. 2023-11-25T05:04:35.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.883+0000 7feb98426640 10 monclient: build_initial_monmap 2023-11-25T05:04:35.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.883+0000 7feb98426640 10 monclient: monmap: 2023-11-25T05:04:35.901 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:35.901 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:35.902 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:35.902 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:35.902 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:35.902 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.902 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:35.902 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:35.902 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:35.902 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.883+0000 7feb98426640 1 librados: starting msgr at 2023-11-25T05:04:35.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.883+0000 7feb98426640 1 librados: starting objecter 2023-11-25T05:04:35.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.883+0000 7feb98426640 1 Processor -- start 2023-11-25T05:04:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.883+0000 7feb98426640 1 -- start start 2023-11-25T05:04:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.883+0000 7feb98426640 1 librados: setting wanted keys 2023-11-25T05:04:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.883+0000 7feb98426640 1 librados: calling monclient init 2023-11-25T05:04:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.883+0000 7feb98426640 10 monclient: init 2023-11-25T05:04:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-25T05:04:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb90070220 0x7feb901cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 10 monclient(hunting): picked mon.noname-c con 0x7feb90070220 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb90072330 0x7feb90139210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 10 monclient(hunting): picked mon.noname-a con 0x7feb90072330 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb900af700 0x7feb90139750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 10 monclient(hunting): picked mon.noname-b con 0x7feb900af700 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb901cc2d0 con 0x7feb90072330 2023-11-25T05:04:35.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7fa229368640 10 monclient(hunting): handle_auth_done global_id 4309 payload 274 2023-11-25T05:04:35.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7fa229368640 1 -- 172.21.15.7:0/928535431 >> v1:172.21.15.90:6789/0 conn(0x7fa2240b2110 legacy=0x7fa22413ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7fa229368640 1 -- 172.21.15.7:0/928535431 >> v1:172.21.15.7:6789/0 conn(0x7fa2240a8c10 legacy=0x7fa22414c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.885+0000 7fa229368640 10 monclient: _finish_hunting 0 2023-11-25T05:04:35.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.885+0000 7fa229368640 1 monclient: found mon.noname-c 2023-11-25T05:04:35.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.885+0000 7fa229368640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:35.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.885+0000 7fa229368640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:35.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.885+0000 7fa229368640 1 -- 172.21.15.7:0/928535431 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2240a9640 con 0x7fa2240a8830 2023-11-25T05:04:35.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb901cd600 con 0x7feb900af700 2023-11-25T05:04:35.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb90139c90 con 0x7feb90070220 2023-11-25T05:04:35.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:35.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.884+0000 7feb98426640 10 monclient(hunting): authenticate will time out at 1402.127734s 2023-11-25T05:04:35.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.885+0000 7feb961c2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb90070220 0x7feb901cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.886+0000 7feb961c2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb90070220 0x7feb901cb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33590/0 (socket says 172.21.15.7:33590) 2023-11-25T05:04:35.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.886+0000 7feb961c2640 1 -- 172.21.15.7:0/400856470 learned_addr learned my addr 172.21.15.7:0/400856470 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:35.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.886+0000 7feb961c2640 10 monclient(hunting): get_auth_request con 0x7feb90070220 auth_method 0 2023-11-25T05:04:35.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.886+0000 7feb961c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.886+0000 7feb961c2640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.886+0000 7feb961c2640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.886+0000 7feb969c3640 1 --2- 172.21.15.7:0/400856470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb900af700 0x7feb90139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.886+0000 7feb969c3640 10 monclient(hunting): get_auth_request con 0x7feb900af700 auth_method 0 2023-11-25T05:04:35.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.886+0000 7feb969c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.886+0000 7feb969c3640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.886+0000 7feb969c3640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.887+0000 7feb961c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.887+0000 7feb961c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.887+0000 7feb961c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.888+0000 7feb969c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.888+0000 7feb969c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.888+0000 7feb969c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.890+0000 7feb961c2640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2023-11-25T05:04:35.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.890+0000 7feb961c2640 1 -- 172.21.15.7:0/400856470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb900af700 msgr2=0x7feb90139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.890+0000 7feb961c2640 1 --2- 172.21.15.7:0/400856470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb900af700 0x7feb90139750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.890+0000 7feb961c2640 1 -- 172.21.15.7:0/400856470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb90072330 msgr2=0x7feb90139210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.890+0000 7feb961c2640 1 --2- 172.21.15.7:0/400856470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb90072330 0x7feb90139210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.890+0000 7feb961c2640 10 monclient: _finish_hunting 0 2023-11-25T05:04:35.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.890+0000 7feb961c2640 1 monclient: found mon.noname-c 2023-11-25T05:04:35.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.890+0000 7feb961c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:35.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.890+0000 7feb961c2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:04:35.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.890+0000 7feb961c2640 1 -- 172.21.15.7:0/400856470 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c0094d0 con 0x7feb90070220 2023-11-25T05:04:35.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.892+0000 7f5f995b0640 10 monclient: get_monmap_and_config 2023-11-25T05:04:35.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.892+0000 7f5f995b0640 10 monclient: build_initial_monmap 2023-11-25T05:04:35.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.892+0000 7f5f995b0640 10 monclient: monmap: 2023-11-25T05:04:35.911 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:35.911 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.911 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:35.893621+0000 2023-11-25T05:04:35.911 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:35.893621+0000 2023-11-25T05:04:35.911 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:35.911 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.911 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:35.911 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:35.912 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:35.912 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:35.912 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:35.912 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.893+0000 7f23cd01f640 10 monclient: get_monmap_and_config 2023-11-25T05:04:35.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.893+0000 7f23cd01f640 10 monclient: build_initial_monmap 2023-11-25T05:04:35.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.893+0000 7f23cd01f640 10 monclient: monmap: 2023-11-25T05:04:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:35.894247+0000 2023-11-25T05:04:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:35.894247+0000 2023-11-25T05:04:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:35.913 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.893+0000 7f5f995b0640 1 Processor -- start 2023-11-25T05:04:35.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.893+0000 7f5f995b0640 1 -- start start 2023-11-25T05:04:35.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.893+0000 7f5f995b0640 10 monclient: init 2023-11-25T05:04:35.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7f5f995b0640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:35.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7f5f995b0640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2023-11-25T05:04:35.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7f5f995b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f940a8a30 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:35.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7f5f995b0640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5f94142bf0 0x7f5f9413ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7f5f995b0640 10 monclient(hunting): picked mon.noname-c con 0x7f5f94142bf0 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:35.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7feb961c2640 1 --2- 172.21.15.7:0/400856470 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb90070220 0x7feb901cb380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009870 tx=0x7feb8c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7f5f995b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f940a8e10 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:35.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7f5f995b0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f940aac10 con 0x7f5f940a8a30 2023-11-25T05:04:35.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7f5f995b0640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f940aaef0 con 0x7f5f940a8e10 2023-11-25T05:04:35.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7f5f995b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7f5f995b0640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f940aaa10 con 0x7f5f94142bf0 2023-11-25T05:04:35.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7f5f995b0640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:35.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7f5f995b0640 10 monclient(hunting): authenticate will time out at 1402.137195s 2023-11-25T05:04:35.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7f23cd01f640 1 Processor -- start 2023-11-25T05:04:35.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7f23cd01f640 1 -- start start 2023-11-25T05:04:35.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7f23cd01f640 10 monclient: init 2023-11-25T05:04:35.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7fa229368640 1 --2- 172.21.15.7:0/928535431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2240a8830 0x7fa2241423e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa2200097a0 tx=0x7fa22002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e225e306c23c4bc server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7feb86ffd640 10 client.?.objecter ms_handle_connect 0x7feb90070220 2023-11-25T05:04:35.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.894+0000 7feb86ffd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:35.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.895+0000 7f23cd01f640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:35.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.895+0000 7f23cd01f640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-11-25T05:04:35.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.895+0000 7f23cd01f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23c80a8a30 0x7f23c81424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.895+0000 7f23cd01f640 10 monclient(hunting): picked mon.noname-b con 0x7f23c80a8a30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:35.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.895+0000 7f23cd01f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23c80a8e10 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:35.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.895+0000 7f23cd01f640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f23c8142a20 0x7f23c8145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f23cd01f640 10 monclient(hunting): picked mon.noname-c con 0x7f23c8142a20 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:35.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f5f927fc640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5f94142bf0 0x7f5f9413ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f23cd01f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c8142eb0 con 0x7f23c80a8e10 2023-11-25T05:04:35.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f23cd01f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f23cd01f640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c80aaa10 con 0x7f23c8142a20 2023-11-25T05:04:35.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f23cd01f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f23cd01f640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80aac10 con 0x7f23c80a8a30 2023-11-25T05:04:35.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f23cd01f640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:35.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f23cd01f640 10 monclient(hunting): authenticate will time out at 1402.139083s 2023-11-25T05:04:35.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f5f937fe640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5f940a8e10 0x7f5f9413f230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:52580/0 (socket says 172.21.15.7:52580) 2023-11-25T05:04:35.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f5f937fe640 1 -- 172.21.15.7:0/3037019212 learned_addr learned my addr 172.21.15.7:0/3037019212 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:35.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f464effd640 1 -- 172.21.15.7:0/64761080 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464402e070 con 0x7f465014a360 2023-11-25T05:04:35.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f5f927fc640 10 monclient(hunting): get_auth_request con 0x7f5f94142bf0 auth_method 0 2023-11-25T05:04:35.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f5f927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f5f927fc640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f5f927fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f464effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f464effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f464effd640 10 monclient: dump: 2023-11-25T05:04:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.922 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f464effd640 10 monclient: _finish_auth 0 2023-11-25T05:04:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f464effd640 10 monclient: _check_auth_tickets 2023-11-25T05:04:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f464effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f465694e640 5 monclient: authenticate success, global_id 4272 2023-11-25T05:04:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f465694e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f464effd640 1 -- 172.21.15.7:0/64761080 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46440315a0 con 0x7f465014a360 2023-11-25T05:04:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f464effd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f465694e640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f465694e640 4 set_mon_vals no callback set 2023-11-25T05:04:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f464effd640 1 -- 172.21.15.7:0/64761080 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4644031930 con 0x7f465014a360 2023-11-25T05:04:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f464effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f464effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.896+0000 7f464effd640 10 monclient: dump: 2023-11-25T05:04:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.924 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.897+0000 7f465694e640 10 monclient: shutdown 2023-11-25T05:04:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.897+0000 7f465694e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:35.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.897+0000 7f5f927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.897+0000 7f465694e640 1 -- 172.21.15.7:0/64761080 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f465014a360 msgr2=0x7f4650151a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.897+0000 7f465694e640 1 --2- 172.21.15.7:0/64761080 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f465014a360 0x7f4650151a70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f46440097a0 tx=0x7f464402b840 comp rx=0 tx=0).stop 2023-11-25T05:04:35.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.897+0000 7f5f927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.897+0000 7f5f927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.897+0000 7f465694e640 1 -- 172.21.15.7:0/64761080 shutdown_connections 2023-11-25T05:04:35.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.897+0000 7f465694e640 1 --2- 172.21.15.7:0/64761080 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f465014a360 0x7f4650151a70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.897+0000 7f465694e640 1 -- 172.21.15.7:0/64761080 >> 172.21.15.7:0/64761080 conn(0x7f46500b01e0 msgr2=0x7f46500b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:35.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.897+0000 7f465694e640 1 -- 172.21.15.7:0/64761080 shutdown_connections 2023-11-25T05:04:35.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.897+0000 7f465694e640 1 -- 172.21.15.7:0/64761080 wait complete. 2023-11-25T05:04:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.898+0000 7f20729ce640 10 monclient: get_monmap_and_config 2023-11-25T05:04:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.898+0000 7f20729ce640 10 monclient: build_initial_monmap 2023-11-25T05:04:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.898+0000 7f20729ce640 10 monclient: monmap: 2023-11-25T05:04:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:35.899053+0000 2023-11-25T05:04:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:35.899053+0000 2023-11-25T05:04:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:35.928 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.898+0000 7f465694e640 10 monclient: build_initial_monmap 2023-11-25T05:04:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.898+0000 7f465694e640 10 monclient: monmap: 2023-11-25T05:04:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:35.929 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:35.929 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.929 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:35.929 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:35.929 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:35.929 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.898+0000 7f23c6d9d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23c80a8a30 0x7f23c81424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f20729ce640 1 Processor -- start 2023-11-25T05:04:35.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f465694e640 1 librados: starting msgr at 2023-11-25T05:04:35.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f465694e640 1 librados: starting objecter 2023-11-25T05:04:35.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f20729ce640 1 -- start start 2023-11-25T05:04:35.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f20729ce640 10 monclient: init 2023-11-25T05:04:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f465694e640 1 Processor -- start 2023-11-25T05:04:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f465694e640 1 -- start start 2023-11-25T05:04:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f465694e640 1 librados: setting wanted keys 2023-11-25T05:04:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f465694e640 1 librados: calling monclient init 2023-11-25T05:04:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f465694e640 10 monclient: init 2023-11-25T05:04:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f23c6d9d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23c80a8a30 0x7f23c81424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33596/0 (socket says 172.21.15.7:33596) 2023-11-25T05:04:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f23c6d9d640 1 -- 172.21.15.7:0/3575006046 learned_addr learned my addr 172.21.15.7:0/3575006046 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f23c6d9d640 10 monclient(hunting): get_auth_request con 0x7f23c80a8a30 auth_method 0 2023-11-25T05:04:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f23c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f23c6d9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f23c6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f5f927fc640 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2023-11-25T05:04:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f5f927fc640 1 -- 172.21.15.7:0/3037019212 >> v1:172.21.15.90:6789/0 conn(0x7f5f940a8e10 legacy=0x7f5f9413f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f5f927fc640 1 -- 172.21.15.7:0/3037019212 >> v1:172.21.15.7:6789/0 conn(0x7f5f940a8a30 legacy=0x7f5f941424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f5f927fc640 10 monclient: _finish_hunting 0 2023-11-25T05:04:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f5f927fc640 1 monclient: found mon.noname-c 2023-11-25T05:04:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f5f927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f5f927fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:35.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.899+0000 7f5f927fc640 1 -- 172.21.15.7:0/3037019212 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f940a97f0 con 0x7f5f94142bf0 2023-11-25T05:04:35.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f20729ce640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:35.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f20729ce640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-11-25T05:04:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-25T05:04:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f465006f2b0 0x7f4650144c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 10 monclient(hunting): picked mon.noname-c con 0x7f465006f2b0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46500af360 0x7f46501db990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 10 monclient(hunting): picked mon.noname-a con 0x7f46500af360 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f465014a360 0x7f465013bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 10 monclient(hunting): picked mon.noname-b con 0x7f465014a360 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46501458b0 con 0x7f46500af360 2023-11-25T05:04:35.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46501cf850 con 0x7f465014a360 2023-11-25T05:04:35.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46501cf4e0 con 0x7f465006f2b0 2023-11-25T05:04:35.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:35.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f465694e640 10 monclient(hunting): authenticate will time out at 1402.143212s 2023-11-25T05:04:35.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f20729ce640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f206c0a8a30 0x7f206c1424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f20729ce640 10 monclient(hunting): picked mon.noname-b con 0x7f206c0a8a30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:35.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f20729ce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f206c0a8e10 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:35.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f20729ce640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f206c142a20 0x7f206c145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f23c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f23c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f23c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f20729ce640 10 monclient(hunting): picked mon.noname-a con 0x7f206c142a20 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:35.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f20729ce640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206c142eb0 con 0x7f206c0a8e10 2023-11-25T05:04:35.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f20729ce640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f20729ce640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f206c0aaa10 con 0x7f206c142a20 2023-11-25T05:04:35.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f20729ce640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f20729ce640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206c0aac10 con 0x7f206c0a8a30 2023-11-25T05:04:35.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f20729ce640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:35.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.900+0000 7f20729ce640 10 monclient(hunting): authenticate will time out at 1402.143064s 2023-11-25T05:04:35.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.901+0000 7f23c759e640 1 --2- 172.21.15.7:0/3575006046 >> v2:172.21.15.90:3300/0 conn(0x7f23c8142a20 0x7f23c8145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.902+0000 7f4654eeb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f465014a360 0x7f465013bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.902+0000 7f23c759e640 10 monclient(hunting): get_auth_request con 0x7f23c8142a20 auth_method 0 2023-11-25T05:04:35.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.902+0000 7f23c759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.902+0000 7f23c759e640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.902+0000 7f23c759e640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.902+0000 7f4654eeb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f465014a360 0x7f465013bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:38332/0 (socket says 172.21.15.7:38332) 2023-11-25T05:04:35.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.902+0000 7f4654eeb640 1 -- 172.21.15.7:0/1972786767 learned_addr learned my addr 172.21.15.7:0/1972786767 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.902+0000 7f4654eeb640 10 monclient(hunting): get_auth_request con 0x7f465014a360 auth_method 0 2023-11-25T05:04:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.902+0000 7f4654eeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.902+0000 7f4654eeb640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.902+0000 7f4654eeb640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.903+0000 7f4654eeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.903+0000 7f4654eeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.903+0000 7f4654eeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.903+0000 7f23c759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.903+0000 7f23c759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.903+0000 7f23c759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.903+0000 7fdac84f2640 10 monclient: get_monmap_and_config 2023-11-25T05:04:35.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.903+0000 7fdac84f2640 10 monclient: build_initial_monmap 2023-11-25T05:04:35.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7fdac84f2640 10 monclient: monmap: 2023-11-25T05:04:35.941 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:35.941 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.941 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:35.904933+0000 2023-11-25T05:04:35.941 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:35.904933+0000 2023-11-25T05:04:35.941 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:35.941 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.942 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:35.942 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:35.942 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:35.942 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:35.942 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:35.942 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f2070f6b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f206c142a20 0x7f206c145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f23c6d9d640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2023-11-25T05:04:35.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f23c6d9d640 1 -- 172.21.15.7:0/3575006046 >> v2:172.21.15.90:3300/0 conn(0x7f23c8142a20 msgr2=0x7f23c8145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:35.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f23c6d9d640 1 --2- 172.21.15.7:0/3575006046 >> v2:172.21.15.90:3300/0 conn(0x7f23c8142a20 0x7f23c8145bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f23c6d9d640 1 -- 172.21.15.7:0/3575006046 >> v1:172.21.15.7:6789/0 conn(0x7f23c80a8e10 legacy=0x7f23c81454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f23c6d9d640 10 monclient: _finish_hunting 0 2023-11-25T05:04:35.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f23c6d9d640 1 monclient: found mon.noname-b 2023-11-25T05:04:35.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f23c6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:35.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f23c6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:04:35.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f23c6d9d640 1 -- 172.21.15.7:0/3575006046 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c80aaef0 con 0x7f23c80a8a30 2023-11-25T05:04:35.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f2070f6b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f206c142a20 0x7f206c145bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40770/0 (socket says 172.21.15.7:40770) 2023-11-25T05:04:35.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f2070f6b640 1 -- 172.21.15.7:0/1790803933 learned_addr learned my addr 172.21.15.7:0/1790803933 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:35.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f2070f6b640 10 monclient(hunting): get_auth_request con 0x7f206c142a20 auth_method 0 2023-11-25T05:04:35.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f2070f6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f2070f6b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f2070f6b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7f464f7fe640 1 --2- 172.21.15.7:0/1972786767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46500af360 0x7f46501db990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7fdac84f2640 1 Processor -- start 2023-11-25T05:04:35.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7fdac84f2640 1 -- start start 2023-11-25T05:04:35.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.904+0000 7fdac84f2640 10 monclient: init 2023-11-25T05:04:35.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.905+0000 7f2070f6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.905+0000 7f2070f6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.905+0000 7f2070f6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.905+0000 7fdac84f2640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:35.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.905+0000 7fdac84f2640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-11-25T05:04:35.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.905+0000 7fdac84f2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdac006f570 0x7fdac013bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.905+0000 7fdac84f2640 10 monclient(hunting): picked mon.noname-b con 0x7fdac006f570 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.905+0000 7fdac84f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdac013c4b0 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.906+0000 7fdac84f2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdac013f220 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.906+0000 7fdac84f2640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac0071550 con 0x7fdac013c4b0 2023-11-25T05:04:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.906+0000 7fdac84f2640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac0071830 con 0x7fdac013f220 2023-11-25T05:04:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.906+0000 7fdac84f2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.906+0000 7fdac84f2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac0071b10 con 0x7fdac006f570 2023-11-25T05:04:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.906+0000 7fdac84f2640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.906+0000 7fdac84f2640 10 monclient(hunting): authenticate will time out at 1402.149005s 2023-11-25T05:04:35.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.907+0000 7f2070f6b640 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2023-11-25T05:04:35.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.907+0000 7f2070f6b640 1 -- 172.21.15.7:0/1790803933 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f206c0a8a30 msgr2=0x7f206c1424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:35.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.907+0000 7f2070f6b640 1 --2- 172.21.15.7:0/1790803933 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f206c0a8a30 0x7f206c1424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.907+0000 7f2070f6b640 1 -- 172.21.15.7:0/1790803933 >> v1:172.21.15.7:6789/0 conn(0x7f206c0a8e10 legacy=0x7f206c1454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:35.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.907+0000 7f2070f6b640 10 monclient: _finish_hunting 0 2023-11-25T05:04:35.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.907+0000 7f2070f6b640 1 monclient: found mon.noname-a 2023-11-25T05:04:35.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.907+0000 7f2070f6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:35.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.907+0000 7f2070f6b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:35.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.907+0000 7f2070f6b640 1 -- 172.21.15.7:0/1790803933 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206c0aaef0 con 0x7f206c142a20 2023-11-25T05:04:35.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.907+0000 7f5f927fc640 1 --2- 172.21.15.7:0/3037019212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f94142bf0 0x7f5f9413ecf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5f840097a0 tx=0x7f5f8402b9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3427200318da8a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.908+0000 7f23c6d9d640 1 --2- 172.21.15.7:0/3575006046 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23c80a8a30 0x7f23c81424e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f23b8009e80 tx=0x7f23b802bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2130818cea50fd74 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.911+0000 7f464f7fe640 10 monclient(hunting): get_auth_request con 0x7f46500af360 auth_method 0 2023-11-25T05:04:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.911+0000 7f464f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.911+0000 7f464f7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.911+0000 7f464f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.911+0000 7f4654eeb640 10 monclient(hunting): handle_auth_done global_id 4321 payload 824 2023-11-25T05:04:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.911+0000 7f4654eeb640 1 -- 172.21.15.7:0/1972786767 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f465006f2b0 msgr2=0x7f4650144c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.911+0000 7f4654eeb640 1 --2- 172.21.15.7:0/1972786767 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f465006f2b0 0x7f4650144c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.911+0000 7f4654eeb640 1 -- 172.21.15.7:0/1972786767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46500af360 msgr2=0x7f46501db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.911+0000 7f4654eeb640 1 --2- 172.21.15.7:0/1972786767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46500af360 0x7f46501db990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.911+0000 7f4654eeb640 10 monclient: _finish_hunting 0 2023-11-25T05:04:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.911+0000 7f4654eeb640 1 monclient: found mon.noname-b 2023-11-25T05:04:35.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.911+0000 7f4654eeb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:35.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.911+0000 7f4654eeb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:35.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.911+0000 7f4654eeb640 1 -- 172.21.15.7:0/1972786767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4644009400 con 0x7f465014a360 2023-11-25T05:04:35.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.912+0000 7fdac628e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdac006f570 0x7fdac013bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.913+0000 7fdac628e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdac006f570 0x7fdac013bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33616/0 (socket says 172.21.15.7:33616) 2023-11-25T05:04:35.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.913+0000 7fdac628e640 1 -- 172.21.15.7:0/1259777803 learned_addr learned my addr 172.21.15.7:0/1259777803 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:35.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.913+0000 7fdac628e640 10 monclient(hunting): get_auth_request con 0x7fdac006f570 auth_method 0 2023-11-25T05:04:35.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.913+0000 7fdac628e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.913+0000 7fdac628e640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.913+0000 7fdac628e640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.913+0000 7fdac628e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.913+0000 7fdac628e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.913+0000 7fdac628e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7f4654eeb640 1 --2- 172.21.15.7:0/1972786767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f465014a360 0x7f465013bfa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f46440097a0 tx=0x7f46440046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7f464cff9640 10 client.?.objecter ms_handle_connect 0x7f465014a360 2023-11-25T05:04:35.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7f464cff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:35.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb86ffd640 1 -- 172.21.15.7:0/400856470 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8c032880 con 0x7feb90070220 2023-11-25T05:04:35.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb86ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb86ffd640 10 monclient: dump: 2023-11-25T05:04:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.954 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb86ffd640 10 monclient: _finish_auth 0 2023-11-25T05:04:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb86ffd640 10 monclient: _check_auth_tickets 2023-11-25T05:04:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb86ffd640 1 -- 172.21.15.7:0/400856470 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8c0329e0 con 0x7feb90070220 2023-11-25T05:04:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb86ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb86ffd640 1 -- 172.21.15.7:0/400856470 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8c032d90 con 0x7feb90070220 2023-11-25T05:04:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb98426640 5 monclient: authenticate success, global_id 4250 2023-11-25T05:04:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb98426640 10 monclient: _renew_subs 2023-11-25T05:04:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb98426640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb98426640 1 -- 172.21.15.7:0/400856470 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb901cc6f0 con 0x7feb90070220 2023-11-25T05:04:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb949bf640 4 set_mon_vals no callback set 2023-11-25T05:04:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb86ffd640 10 monclient: dump: 2023-11-25T05:04:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.956 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.956 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.956 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.956 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.956 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.956 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.956 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.956 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.956 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb98426640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:35.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb98426640 10 monclient: _renew_subs 2023-11-25T05:04:35.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb98426640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:35.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb98426640 1 -- 172.21.15.7:0/400856470 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb9013b050 con 0x7feb90070220 2023-11-25T05:04:35.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.914+0000 7feb98426640 1 librados: init done 2023-11-25T05:04:35.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.915+0000 7feb98426640 10 client.4250.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:35.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.915+0000 7feb98426640 15 client.4250.objecter _session_command_op_assign -1 1 2023-11-25T05:04:35.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.915+0000 7feb98426640 20 client.4250.objecter _get_session osd=-1 returning homeless 2023-11-25T05:04:35.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.915+0000 7feb98426640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:35.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.915+0000 7feb98426640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:35.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.915+0000 7feb98426640 1 -- 172.21.15.7:0/400856470 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feb9013bd50 con 0x7feb90070220 2023-11-25T05:04:35.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.916+0000 7f2070f6b640 1 --2- 172.21.15.7:0/1790803933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f206c142a20 0x7f206c145bb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f205c0098a0 tx=0x7f205c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcda255038432ce server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.916+0000 7fdac628e640 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2023-11-25T05:04:35.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.916+0000 7fdac628e640 1 -- 172.21.15.7:0/1259777803 >> v1:172.21.15.90:6789/0 conn(0x7fdac013f220 legacy=0x7fdac0149360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.916+0000 7fdac628e640 1 -- 172.21.15.7:0/1259777803 >> v1:172.21.15.7:6789/0 conn(0x7fdac013c4b0 legacy=0x7fdac013eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:35.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.916+0000 7fdac628e640 10 monclient: _finish_hunting 0 2023-11-25T05:04:35.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.916+0000 7fdac628e640 1 monclient: found mon.noname-b 2023-11-25T05:04:35.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.916+0000 7fdac628e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:35.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.916+0000 7fdac628e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:04:35.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.916+0000 7fdac628e640 1 -- 172.21.15.7:0/1259777803 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac00aecb0 con 0x7fdac006f570 2023-11-25T05:04:35.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.919+0000 7fdac628e640 1 --2- 172.21.15.7:0/1259777803 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdac006f570 0x7fdac013bf70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00bb40 tx=0x7fdabc02e0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd839e3a4206ce0f server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f206affd640 1 -- 172.21.15.7:0/1790803933 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c031640 con 0x7f206c142a20 2023-11-25T05:04:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f206affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f206affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f206affd640 10 monclient: dump: 2023-11-25T05:04:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.964 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f206affd640 10 monclient: _finish_auth 0 2023-11-25T05:04:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f206affd640 10 monclient: _check_auth_tickets 2023-11-25T05:04:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f206affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f206affd640 1 -- 172.21.15.7:0/1790803933 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205c0317a0 con 0x7f206c142a20 2023-11-25T05:04:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f206affd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f206affd640 1 -- 172.21.15.7:0/1790803933 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c031b30 con 0x7f206c142a20 2023-11-25T05:04:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f20729ce640 5 monclient: authenticate success, global_id 4281 2023-11-25T05:04:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f20729ce640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f20729ce640 4 set_mon_vals no callback set 2023-11-25T05:04:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f206affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f206affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.922+0000 7f206affd640 10 monclient: dump: 2023-11-25T05:04:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.966 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.923+0000 7f20729ce640 10 monclient: shutdown 2023-11-25T05:04:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.923+0000 7f20729ce640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.923+0000 7f20729ce640 1 -- 172.21.15.7:0/1790803933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f206c142a20 msgr2=0x7f206c145bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.923+0000 7f20729ce640 1 --2- 172.21.15.7:0/1790803933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f206c142a20 0x7f206c145bb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f205c0098a0 tx=0x7f205c02b8f0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.923+0000 7f20729ce640 1 -- 172.21.15.7:0/1790803933 shutdown_connections 2023-11-25T05:04:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.923+0000 7f20729ce640 1 --2- 172.21.15.7:0/1790803933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f206c142a20 0x7f206c145bb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.923+0000 7f20729ce640 1 --2- 172.21.15.7:0/1790803933 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f206c0a8a30 0x7f206c1424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.923+0000 7f20729ce640 1 -- 172.21.15.7:0/1790803933 >> 172.21.15.7:0/1790803933 conn(0x7f206c061a30 msgr2=0x7f206c0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.923+0000 7f20729ce640 1 -- 172.21.15.7:0/1790803933 shutdown_connections 2023-11-25T05:04:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.923+0000 7f20729ce640 1 -- 172.21.15.7:0/1790803933 wait complete. 2023-11-25T05:04:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.924+0000 7f20729ce640 10 monclient: build_initial_monmap 2023-11-25T05:04:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.924+0000 7f20729ce640 10 monclient: monmap: 2023-11-25T05:04:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:35.968 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.924+0000 7f20729ce640 1 librados: starting msgr at 2023-11-25T05:04:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.924+0000 7f20729ce640 1 librados: starting objecter 2023-11-25T05:04:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.924+0000 7f20729ce640 1 Processor -- start 2023-11-25T05:04:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.924+0000 7f20729ce640 1 -- start start 2023-11-25T05:04:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.924+0000 7f20729ce640 1 librados: setting wanted keys 2023-11-25T05:04:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.924+0000 7f20729ce640 1 librados: calling monclient init 2023-11-25T05:04:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.924+0000 7f20729ce640 10 monclient: init 2023-11-25T05:04:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:04:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f206c0a8a30 0x7f206c0af050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 10 monclient(hunting): picked mon.noname-c con 0x7f206c0a8a30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f206c0a8e10 0x7f206c0afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 10 monclient(hunting): picked mon.noname-b con 0x7f206c0a8e10 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f206c142a20 0x7f206c13b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 10 monclient(hunting): picked mon.noname-a con 0x7f206c142a20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c0adfd0 con 0x7f206c142a20 2023-11-25T05:04:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c0b0000 con 0x7f206c0a8e10 2023-11-25T05:04:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206c0b0170 con 0x7f206c0a8a30 2023-11-25T05:04:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.925+0000 7f20729ce640 10 monclient(hunting): authenticate will time out at 1402.168638s 2023-11-25T05:04:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.927+0000 7f2070f6b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f206c142a20 0x7f206c13b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.927+0000 7f2070f6b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f206c142a20 0x7f206c13b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40786/0 (socket says 172.21.15.7:40786) 2023-11-25T05:04:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.927+0000 7f2070f6b640 1 -- 172.21.15.7:0/582799257 learned_addr learned my addr 172.21.15.7:0/582799257 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.927+0000 7f2070f6b640 10 monclient(hunting): get_auth_request con 0x7f206c142a20 auth_method 0 2023-11-25T05:04:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.927+0000 7f2070f6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.927+0000 7f2070f6b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.927+0000 7f2070f6b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.928+0000 7f2070f6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.928+0000 7f2070f6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.928+0000 7f2070f6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.930+0000 7f2070f6b640 10 monclient(hunting): handle_auth_done global_id 4287 payload 824 2023-11-25T05:04:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.930+0000 7f2070f6b640 1 -- 172.21.15.7:0/582799257 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f206c0a8a30 msgr2=0x7f206c0af050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.930+0000 7f2070f6b640 1 --2- 172.21.15.7:0/582799257 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f206c0a8a30 0x7f206c0af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.930+0000 7f2070f6b640 1 -- 172.21.15.7:0/582799257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f206c0a8e10 msgr2=0x7f206c0afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.930+0000 7f2070f6b640 1 --2- 172.21.15.7:0/582799257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f206c0a8e10 0x7f206c0afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.930+0000 7f2070f6b640 10 monclient: _finish_hunting 0 2023-11-25T05:04:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.930+0000 7f2070f6b640 1 monclient: found mon.noname-a 2023-11-25T05:04:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.930+0000 7f2070f6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.930+0000 7f2070f6b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.930+0000 7f2070f6b640 1 -- 172.21.15.7:0/582799257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205c0094d0 con 0x7f206c142a20 2023-11-25T05:04:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.933+0000 7f2070f6b640 1 --2- 172.21.15.7:0/582799257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f206c142a20 0x7f206c13b880 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f205c034040 tx=0x7f205c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.933+0000 7f2068ff9640 10 client.?.objecter ms_handle_connect 0x7f206c142a20 2023-11-25T05:04:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.933+0000 7f2068ff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f2068ff9640 1 -- 172.21.15.7:0/582799257 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c03abc0 con 0x7f206c142a20 2023-11-25T05:04:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f2068ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f2068ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f2068ff9640 10 monclient: dump: 2023-11-25T05:04:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.972 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f2068ff9640 10 monclient: _finish_auth 0 2023-11-25T05:04:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f2068ff9640 10 monclient: _check_auth_tickets 2023-11-25T05:04:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f2068ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f2068ff9640 1 -- 172.21.15.7:0/582799257 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205c03ad20 con 0x7f206c142a20 2023-11-25T05:04:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f20729ce640 5 monclient: authenticate success, global_id 4287 2023-11-25T05:04:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f20729ce640 10 monclient: _renew_subs 2023-11-25T05:04:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f20729ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f20729ce640 1 -- 172.21.15.7:0/582799257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206c13bdc0 con 0x7f206c142a20 2023-11-25T05:04:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f2068ff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f2068ff9640 1 -- 172.21.15.7:0/582799257 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c044630 con 0x7f206c142a20 2023-11-25T05:04:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f2068ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f2068ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f2068ff9640 10 monclient: dump: 2023-11-25T05:04:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.974 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f206a7fc640 4 set_mon_vals no callback set 2023-11-25T05:04:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f20729ce640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f20729ce640 10 monclient: _renew_subs 2023-11-25T05:04:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f20729ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f20729ce640 1 -- 172.21.15.7:0/582799257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f206c13c1b0 con 0x7f206c142a20 2023-11-25T05:04:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.937+0000 7f20729ce640 1 librados: init done 2023-11-25T05:04:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.938+0000 7f20729ce640 10 client.4287.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.938+0000 7f20729ce640 15 client.4287.objecter _session_command_op_assign -1 1 2023-11-25T05:04:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.938+0000 7f20729ce640 20 client.4287.objecter _get_session osd=-1 returning homeless 2023-11-25T05:04:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.938+0000 7f20729ce640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.938+0000 7f20729ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.938+0000 7f20729ce640 1 -- 172.21.15.7:0/582799257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f206c141fc0 con 0x7f206c142a20 2023-11-25T05:04:35.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.943+0000 7feb86ffd640 1 -- 172.21.15.7:0/400856470 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7feb8c04d3f0 con 0x7feb90070220 2023-11-25T05:04:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.943+0000 7feb86ffd640 1 --2- 172.21.15.7:0/400856470 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7feb7406c820 0x7feb7406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.944+0000 7feb959c1640 1 --2- 172.21.15.7:0/400856470 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7feb7406c820 0x7feb7406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.944+0000 7feb959c1640 10 monclient: get_auth_request con 0x7feb7406c820 auth_method 0 2023-11-25T05:04:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.944+0000 7feb959c1640 1 --2- 172.21.15.7:0/400856470 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7feb7406c820 0x7feb7406ece0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feb88007fd0 tx=0x7feb88007d40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.944+0000 7feb86ffd640 10 client.4250.objecter ms_handle_connect 0x7feb7406c820 2023-11-25T05:04:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 1 -- 172.21.15.7:0/400856470 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7feb8c07a960 con 0x7feb90070220 2023-11-25T05:04:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 10 client.4250.objecter ms_dispatch 0x7feb90000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 3 client.4250.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 3 client.4250.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 10 client.4250.objecter checking command 1 2023-11-25T05:04:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 1 --2- 172.21.15.7:0/400856470 >> [v2:172.21.15.7:6816/59754388,v1:172.21.15.7:6817/59754388] conn(0x7feb74072c00 0x7feb74075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 20 client.4250.objecter _get_session s=0x7feb74072530 osd=3 3 2023-11-25T05:04:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 20 client.4250.objecter put_session s=0x7feb74072530 osd=3 3 2023-11-25T05:04:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 15 client.4250.objecter _session_command_op_remove -1 1 2023-11-25T05:04:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 20 client.4250.objecter _get_session s=0x7feb74072530 osd=3 3 2023-11-25T05:04:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 20 client.4250.objecter get_session s=0x7feb74072530 osd=3 3 2023-11-25T05:04:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 15 client.4250.objecter _session_command_op_assign 3 1 2023-11-25T05:04:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 20 client.4250.objecter put_session s=0x7feb74072530 osd=3 4 2023-11-25T05:04:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 10 client.4250.objecter _send_command 1 2023-11-25T05:04:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 1 -- 172.21.15.7:0/400856470 --> [v2:172.21.15.7:6816/59754388,v1:172.21.15.7:6817/59754388] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb740756d0 con 0x7feb74072c00 2023-11-25T05:04:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 20 client.4250.objecter dump_active .. 0 homeless 2023-11-25T05:04:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 1 -- 172.21.15.7:0/400856470 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb8c0765a0 con 0x7feb90070220 2023-11-25T05:04:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.948+0000 7feb86ffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-11-25T05:04:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.949+0000 7feb969c3640 1 --2- 172.21.15.7:0/400856470 >> [v2:172.21.15.7:6816/59754388,v1:172.21.15.7:6817/59754388] conn(0x7feb74072c00 0x7feb74075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.949+0000 7feb969c3640 10 monclient: get_auth_request con 0x7feb74072c00 auth_method 0 2023-11-25T05:04:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23c659c640 1 -- 172.21.15.7:0/3575006046 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b802cae0 con 0x7f23c80a8a30 2023-11-25T05:04:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23c659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23c659c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23c659c640 10 monclient: dump: 2023-11-25T05:04:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.981 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23c659c640 10 monclient: _finish_auth 0 2023-11-25T05:04:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23c659c640 10 monclient: _check_auth_tickets 2023-11-25T05:04:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23c659c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23c659c640 1 -- 172.21.15.7:0/3575006046 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b802cc40 con 0x7f23c80a8a30 2023-11-25T05:04:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23c659c640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23c659c640 1 -- 172.21.15.7:0/3575006046 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b80343f0 con 0x7f23c80a8a30 2023-11-25T05:04:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23c659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23c659c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23c659c640 10 monclient: dump: 2023-11-25T05:04:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.982 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23cd01f640 5 monclient: authenticate success, global_id 4253 2023-11-25T05:04:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23cd01f640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.952+0000 7f23cd01f640 4 set_mon_vals no callback set 2023-11-25T05:04:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.953+0000 7f23cd01f640 10 monclient: shutdown 2023-11-25T05:04:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.953+0000 7f23cd01f640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.953+0000 7f23cd01f640 1 -- 172.21.15.7:0/3575006046 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23c80a8a30 msgr2=0x7f23c81424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.953+0000 7f23cd01f640 1 --2- 172.21.15.7:0/3575006046 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23c80a8a30 0x7f23c81424e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f23b8009e80 tx=0x7f23b802bf40 comp rx=0 tx=0).stop 2023-11-25T05:04:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.953+0000 7f23cd01f640 1 -- 172.21.15.7:0/3575006046 shutdown_connections 2023-11-25T05:04:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.953+0000 7f23cd01f640 1 --2- 172.21.15.7:0/3575006046 >> v2:172.21.15.90:3300/0 conn(0x7f23c8142a20 0x7f23c8145bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.953+0000 7f23cd01f640 1 --2- 172.21.15.7:0/3575006046 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23c80a8a30 0x7f23c81424e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.953+0000 7f23cd01f640 1 -- 172.21.15.7:0/3575006046 >> 172.21.15.7:0/3575006046 conn(0x7f23c8061a30 msgr2=0x7f23c80b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.953+0000 7f23cd01f640 1 -- 172.21.15.7:0/3575006046 shutdown_connections 2023-11-25T05:04:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.953+0000 7f23cd01f640 1 -- 172.21.15.7:0/3575006046 wait complete. 2023-11-25T05:04:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.954+0000 7feb969c3640 1 --2- 172.21.15.7:0/400856470 >> [v2:172.21.15.7:6816/59754388,v1:172.21.15.7:6817/59754388] conn(0x7feb74072c00 0x7feb74075020 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.954+0000 7feb86ffd640 10 client.4250.objecter ms_handle_connect 0x7feb74072c00 2023-11-25T05:04:35.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.954+0000 7f23cd01f640 10 monclient: build_initial_monmap 2023-11-25T05:04:35.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.954+0000 7f23cd01f640 10 monclient: monmap: 2023-11-25T05:04:35.985 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:35.985 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:35.986 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.954+0000 7f23cd01f640 1 librados: starting msgr at 2023-11-25T05:04:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.954+0000 7f23cd01f640 1 librados: starting objecter 2023-11-25T05:04:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.954+0000 7f23cd01f640 1 Processor -- start 2023-11-25T05:04:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.955+0000 7f23cd01f640 1 -- start start 2023-11-25T05:04:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.955+0000 7f23cd01f640 1 librados: setting wanted keys 2023-11-25T05:04:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.955+0000 7f23cd01f640 1 librados: calling monclient init 2023-11-25T05:04:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.955+0000 7f23cd01f640 10 monclient: init 2023-11-25T05:04:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.955+0000 7f23cd01f640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.955+0000 7f23cd01f640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-25T05:04:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.955+0000 7f23cd01f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23c80a8a30 0x7f23c80aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.955+0000 7f23cd01f640 10 monclient(hunting): picked mon.noname-a con 0x7f23c80a8a30 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.955+0000 7f23cd01f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23c80a8e10 0x7f23c813b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.955+0000 7f23cd01f640 10 monclient(hunting): picked mon.noname-b con 0x7f23c80a8e10 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.955+0000 7f23cd01f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23c8142a20 0x7f23c813bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.956+0000 7f23cd01f640 10 monclient(hunting): picked mon.noname-c con 0x7f23c8142a20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.956+0000 7f23cd01f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.956+0000 7f23cd01f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80b0ff0 con 0x7f23c80a8a30 2023-11-25T05:04:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.956+0000 7f23cd01f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.956+0000 7f23cd01f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80afd70 con 0x7f23c80a8e10 2023-11-25T05:04:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.956+0000 7f23cd01f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.956+0000 7f23cd01f640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80add30 con 0x7f23c8142a20 2023-11-25T05:04:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.956+0000 7f23cd01f640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.956+0000 7f23cd01f640 10 monclient(hunting): authenticate will time out at 1402.198950s 2023-11-25T05:04:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.957+0000 7fdac528c640 1 -- 172.21.15.7:0/1259777803 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdabc02ee70 con 0x7fdac006f570 2023-11-25T05:04:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.957+0000 7fdac528c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.957+0000 7fdac528c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.957+0000 7fdac528c640 10 monclient: dump: 2023-11-25T05:04:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.990 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.957+0000 7fdac528c640 10 monclient: _finish_auth 0 2023-11-25T05:04:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.957+0000 7fdac528c640 10 monclient: _check_auth_tickets 2023-11-25T05:04:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.957+0000 7fdac528c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.957+0000 7fdac528c640 1 -- 172.21.15.7:0/1259777803 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdabc036440 con 0x7fdac006f570 2023-11-25T05:04:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.957+0000 7fdac528c640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.957+0000 7fdac528c640 1 -- 172.21.15.7:0/1259777803 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdabc0367d0 con 0x7fdac006f570 2023-11-25T05:04:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.957+0000 7fdac528c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.957+0000 7fdac528c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.957+0000 7fdac528c640 10 monclient: dump: 2023-11-25T05:04:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:35.991 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.958+0000 7fdac84f2640 5 monclient: authenticate success, global_id 4256 2023-11-25T05:04:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.958+0000 7fdac84f2640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.958+0000 7fdac84f2640 4 set_mon_vals no callback set 2023-11-25T05:04:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.958+0000 7f23c759e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23c8142a20 0x7f23c813bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.958+0000 7fdac84f2640 10 monclient: shutdown 2023-11-25T05:04:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.958+0000 7fdac84f2640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.958+0000 7fdac84f2640 1 -- 172.21.15.7:0/1259777803 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdac006f570 msgr2=0x7fdac013bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.958+0000 7fdac84f2640 1 --2- 172.21.15.7:0/1259777803 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdac006f570 0x7fdac013bf70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00bb40 tx=0x7fdabc02e0c0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.959+0000 7fdac84f2640 1 -- 172.21.15.7:0/1259777803 shutdown_connections 2023-11-25T05:04:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.959+0000 7fdac84f2640 1 --2- 172.21.15.7:0/1259777803 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdac006f570 0x7fdac013bf70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.959+0000 7fdac84f2640 1 -- 172.21.15.7:0/1259777803 >> 172.21.15.7:0/1259777803 conn(0x7fdac006c100 msgr2=0x7fdac00b4660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.959+0000 7fdac84f2640 1 -- 172.21.15.7:0/1259777803 shutdown_connections 2023-11-25T05:04:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.959+0000 7fdac84f2640 1 -- 172.21.15.7:0/1259777803 wait complete. 2023-11-25T05:04:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.960+0000 7fdac84f2640 10 monclient: build_initial_monmap 2023-11-25T05:04:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.960+0000 7fdac84f2640 10 monclient: monmap: 2023-11-25T05:04:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:35.994 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.960+0000 7fdac84f2640 1 librados: starting msgr at 2023-11-25T05:04:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.960+0000 7fdac84f2640 1 librados: starting objecter 2023-11-25T05:04:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.961+0000 7fdac84f2640 1 Processor -- start 2023-11-25T05:04:35.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.961+0000 7f2068ff9640 1 -- 172.21.15.7:0/582799257 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f205c02f050 con 0x7f206c142a20 2023-11-25T05:04:35.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.961+0000 7fdac84f2640 1 -- start start 2023-11-25T05:04:35.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.961+0000 7fdac84f2640 1 librados: setting wanted keys 2023-11-25T05:04:35.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.961+0000 7fdac84f2640 1 librados: calling monclient init 2023-11-25T05:04:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.961+0000 7fdac84f2640 10 monclient: init 2023-11-25T05:04:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.961+0000 7f2068ff9640 1 --2- 172.21.15.7:0/582799257 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f205806c820 0x7f205806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.961+0000 7f206bfff640 1 --2- 172.21.15.7:0/582799257 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f205806c820 0x7f205806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7f206bfff640 10 monclient: get_auth_request con 0x7f205806c820 auth_method 0 2023-11-25T05:04:35.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:35.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:04:35.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdac006f570 0x7fdac01407f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:35.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7f2068ff9640 10 client.4287.objecter ms_handle_connect 0x7f205806c820 2023-11-25T05:04:35.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7f206bfff640 1 --2- 172.21.15.7:0/582799257 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f205806c820 0x7f205806ece0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f206400ab90 tx=0x7f2064009250 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 10 monclient(hunting): picked mon.noname-a con 0x7fdac006f570 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdac013c4b0 0x7fdac013cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 10 monclient(hunting): picked mon.noname-c con 0x7fdac013c4b0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdac013f220 0x7fdac013d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 10 monclient(hunting): picked mon.noname-b con 0x7fdac013f220 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac013d6f0 con 0x7fdac006f570 2023-11-25T05:04:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac01415f0 con 0x7fdac013f220 2023-11-25T05:04:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac0142c50 con 0x7fdac013c4b0 2023-11-25T05:04:36.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7fdac84f2640 10 monclient(hunting): authenticate will time out at 1402.205650s 2023-11-25T05:04:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7f23c759e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23c8142a20 0x7f23c813bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33646/0 (socket says 172.21.15.7:33646) 2023-11-25T05:04:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7f23c759e640 1 -- 172.21.15.7:0/412701693 learned_addr learned my addr 172.21.15.7:0/412701693 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7f23c759e640 10 monclient(hunting): get_auth_request con 0x7f23c8142a20 auth_method 0 2023-11-25T05:04:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7f23c759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.962+0000 7f23c759e640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.963+0000 7f23c759e640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.964+0000 7f23c759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.964+0000 7f23c759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.964+0000 7f23c759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 1 -- 172.21.15.7:0/582799257 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f205c07ad50 con 0x7f206c142a20 2023-11-25T05:04:36.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 10 client.4287.objecter ms_dispatch 0x7f206c000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:36.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 3 client.4287.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:36.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 3 client.4287.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:36.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 10 client.4287.objecter checking command 1 2023-11-25T05:04:36.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 1 --2- 172.21.15.7:0/582799257 >> [v2:172.21.15.7:6824/2271997805,v1:172.21.15.7:6825/2271997805] conn(0x7f2058072c00 0x7f2058075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 20 client.4287.objecter _get_session s=0x7f2058072530 osd=0 3 2023-11-25T05:04:36.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 20 client.4287.objecter put_session s=0x7f2058072530 osd=0 3 2023-11-25T05:04:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 15 client.4287.objecter _session_command_op_remove -1 1 2023-11-25T05:04:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 20 client.4287.objecter _get_session s=0x7f2058072530 osd=0 3 2023-11-25T05:04:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 20 client.4287.objecter get_session s=0x7f2058072530 osd=0 3 2023-11-25T05:04:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 15 client.4287.objecter _session_command_op_assign 0 1 2023-11-25T05:04:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 20 client.4287.objecter put_session s=0x7f2058072530 osd=0 4 2023-11-25T05:04:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 10 client.4287.objecter _send_command 1 2023-11-25T05:04:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 1 -- 172.21.15.7:0/582799257 --> [v2:172.21.15.7:6824/2271997805,v1:172.21.15.7:6825/2271997805] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20580756d0 con 0x7f2058072c00 2023-11-25T05:04:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 20 client.4287.objecter dump_active .. 0 homeless 2023-11-25T05:04:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 1 -- 172.21.15.7:0/582799257 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f205c076990 con 0x7f206c142a20 2023-11-25T05:04:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.965+0000 7f2068ff9640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-11-25T05:04:36.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.966+0000 7f206b7fe640 1 --2- 172.21.15.7:0/582799257 >> [v2:172.21.15.7:6824/2271997805,v1:172.21.15.7:6825/2271997805] conn(0x7f2058072c00 0x7f2058075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.966+0000 7f206b7fe640 10 monclient: get_auth_request con 0x7f2058072c00 auth_method 0 2023-11-25T05:04:36.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.966+0000 7fdac628e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdac006f570 0x7fdac01407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.967+0000 7f23c6d9d640 1 --2- 172.21.15.7:0/412701693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23c80a8a30 0x7f23c80aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.967+0000 7fdac628e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdac006f570 0x7fdac01407f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40804/0 (socket says 172.21.15.7:40804) 2023-11-25T05:04:36.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.967+0000 7fdac628e640 1 -- 172.21.15.7:0/2141132124 learned_addr learned my addr 172.21.15.7:0/2141132124 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.967+0000 7fdac628e640 10 monclient(hunting): get_auth_request con 0x7fdac006f570 auth_method 0 2023-11-25T05:04:36.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.967+0000 7fdac628e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.967+0000 7fdac628e640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.967+0000 7fdac628e640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.968+0000 7f23c6d9d640 10 monclient(hunting): get_auth_request con 0x7f23c80a8a30 auth_method 0 2023-11-25T05:04:36.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.968+0000 7f23c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.968+0000 7f23c6d9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.968+0000 7f23c6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.968+0000 7f23c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.968+0000 7f23c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.968+0000 7f23c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.969+0000 7f23c759e640 10 monclient(hunting): handle_auth_done global_id 4262 payload 824 2023-11-25T05:04:36.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.969+0000 7f23c759e640 1 -- 172.21.15.7:0/412701693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23c80a8e10 msgr2=0x7f23c813b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.969+0000 7f23c759e640 1 --2- 172.21.15.7:0/412701693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23c80a8e10 0x7f23c813b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.969+0000 7f23c759e640 1 -- 172.21.15.7:0/412701693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23c80a8a30 msgr2=0x7f23c80aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.969+0000 7f23c759e640 1 --2- 172.21.15.7:0/412701693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23c80a8a30 0x7f23c80aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.969+0000 7f23c759e640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.969+0000 7f23c759e640 1 monclient: found mon.noname-c 2023-11-25T05:04:36.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.969+0000 7f23c759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.969+0000 7f23c759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.969+0000 7f23c759e640 1 -- 172.21.15.7:0/412701693 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b8009b30 con 0x7f23c8142a20 2023-11-25T05:04:36.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.969+0000 7fdac628e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.969+0000 7fdac628e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.969+0000 7fdac628e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.970+0000 7f2068ff9640 10 client.4287.objecter ms_handle_connect 0x7f2058072c00 2023-11-25T05:04:36.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.970+0000 7f206b7fe640 1 --2- 172.21.15.7:0/582799257 >> [v2:172.21.15.7:6824/2271997805,v1:172.21.15.7:6825/2271997805] conn(0x7f2058072c00 0x7f2058075020 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.977+0000 7fdac628e640 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2023-11-25T05:04:36.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.977+0000 7fdac628e640 1 -- 172.21.15.7:0/2141132124 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdac013c4b0 msgr2=0x7fdac013cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.977+0000 7fdac628e640 1 --2- 172.21.15.7:0/2141132124 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdac013c4b0 0x7fdac013cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.977+0000 7fdac628e640 1 -- 172.21.15.7:0/2141132124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdac013f220 msgr2=0x7fdac013d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.977+0000 7fdac628e640 1 --2- 172.21.15.7:0/2141132124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdac013f220 0x7fdac013d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.977+0000 7fdac628e640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.977+0000 7fdac628e640 1 monclient: found mon.noname-a 2023-11-25T05:04:36.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.977+0000 7fdac628e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.977+0000 7fdac628e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.977+0000 7fdac628e640 1 -- 172.21.15.7:0/2141132124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdabc00b770 con 0x7fdac006f570 2023-11-25T05:04:36.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.982+0000 7fdac628e640 1 --2- 172.21.15.7:0/2141132124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdac006f570 0x7fdac01407f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdabc000c00 tx=0x7fdabc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.982+0000 7fdab6ffd640 10 client.?.objecter ms_handle_connect 0x7fdac006f570 2023-11-25T05:04:36.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.982+0000 7fdab6ffd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:36.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.986+0000 7fdab6ffd640 1 -- 172.21.15.7:0/2141132124 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdabc041b10 con 0x7fdac006f570 2023-11-25T05:04:36.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.986+0000 7fdab6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.986+0000 7fdab6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:36.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.986+0000 7fdab6ffd640 10 monclient: dump: 2023-11-25T05:04:36.018 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.019 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.019 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.019 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.019 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.019 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.019 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.019 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.019 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.019 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.986+0000 7fdab6ffd640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.986+0000 7fdab6ffd640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.986+0000 7fdab6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.986+0000 7fdab6ffd640 1 -- 172.21.15.7:0/2141132124 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdabc041c90 con 0x7fdac006f570 2023-11-25T05:04:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.986+0000 7fdab6ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.986+0000 7fdac4a8b640 4 set_mon_vals no callback set 2023-11-25T05:04:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.987+0000 7fdab6ffd640 1 -- 172.21.15.7:0/2141132124 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdabc0345f0 con 0x7fdac006f570 2023-11-25T05:04:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.987+0000 7fdab6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.987+0000 7fdab6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.987+0000 7fdab6ffd640 10 monclient: dump: 2023-11-25T05:04:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.021 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.987+0000 7fdac84f2640 5 monclient: authenticate success, global_id 4293 2023-11-25T05:04:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.987+0000 7fdac84f2640 10 monclient: _renew_subs 2023-11-25T05:04:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.987+0000 7fdac84f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.987+0000 7fdac84f2640 1 -- 172.21.15.7:0/2141132124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac013e860 con 0x7fdac006f570 2023-11-25T05:04:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.987+0000 7fdac84f2640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.987+0000 7fdac84f2640 10 monclient: _renew_subs 2023-11-25T05:04:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.987+0000 7fdac84f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.987+0000 7fdac84f2640 1 -- 172.21.15.7:0/2141132124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac0140d30 con 0x7fdac006f570 2023-11-25T05:04:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.987+0000 7fdac84f2640 1 librados: init done 2023-11-25T05:04:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.988+0000 7fdac84f2640 10 client.4293.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.989+0000 7fdac84f2640 15 client.4293.objecter _session_command_op_assign -1 1 2023-11-25T05:04:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.989+0000 7fdac84f2640 20 client.4293.objecter _get_session osd=-1 returning homeless 2023-11-25T05:04:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.989+0000 7fdac84f2640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.989+0000 7fdac84f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:35.989+0000 7fdac84f2640 1 -- 172.21.15.7:0/2141132124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdac013f690 con 0x7fdac006f570 2023-11-25T05:04:36.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.001+0000 7f23c759e640 1 --2- 172.21.15.7:0/412701693 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23c8142a20 0x7f23c813bde0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f23c000e820 tx=0x7f23c000ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.001+0000 7f23bffff640 10 client.?.objecter ms_handle_connect 0x7f23c8142a20 2023-11-25T05:04:36.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.001+0000 7f23bffff640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:36.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.009+0000 7feb86ffd640 1 -- 172.21.15.7:0/400856470 <== osd.3 v2:172.21.15.7:6816/59754388 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7feb740756d0 con 0x7feb74072c00 2023-11-25T05:04:36.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.009+0000 7feb86ffd640 10 client.4250.objecter ms_dispatch 0x7feb90000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:04:36.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.009+0000 7feb86ffd640 10 client.4250.objecter _finish_command 1 = system:0 2023-11-25T05:04:36.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.009+0000 7feb86ffd640 20 client.4250.objecter put_session s=0x7feb74072530 osd=3 4 2023-11-25T05:04:36.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.009+0000 7feb86ffd640 15 client.4250.objecter _session_command_op_remove 3 1 2023-11-25T05:04:36.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.010+0000 7fdab6ffd640 1 -- 172.21.15.7:0/2141132124 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdabc03d040 con 0x7fdac006f570 2023-11-25T05:04:36.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.010+0000 7fdab6ffd640 1 --2- 172.21.15.7:0/2141132124 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdaa406c850 0x7fdaa406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.013+0000 7fdac5a8d640 1 --2- 172.21.15.7:0/2141132124 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdaa406c850 0x7fdaa406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.013+0000 7fdac5a8d640 10 monclient: get_auth_request con 0x7fdaa406c850 auth_method 0 2023-11-25T05:04:36.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.013+0000 7fdab6ffd640 1 -- 172.21.15.7:0/2141132124 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdabc07f760 con 0x7fdac006f570 2023-11-25T05:04:36.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.013+0000 7fdab6ffd640 10 client.4293.objecter ms_dispatch 0x7fdac0000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:36.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.013+0000 7fdab6ffd640 3 client.4293.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.013+0000 7fdab6ffd640 3 client.4293.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.014+0000 7fdab6ffd640 10 client.4293.objecter checking command 1 2023-11-25T05:04:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.014+0000 7fdab6ffd640 1 --2- 172.21.15.7:0/2141132124 >> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] conn(0x7fdaa4072ac0 0x7fdaa4074ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.014+0000 7fdab6ffd640 20 client.4293.objecter _get_session s=0x7fdabc041c90 osd=7 3 2023-11-25T05:04:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.014+0000 7fdab6ffd640 20 client.4293.objecter put_session s=0x7fdabc041c90 osd=7 3 2023-11-25T05:04:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.014+0000 7fdab6ffd640 15 client.4293.objecter _session_command_op_remove -1 1 2023-11-25T05:04:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.014+0000 7fdab6ffd640 20 client.4293.objecter _get_session s=0x7fdabc041c90 osd=7 3 2023-11-25T05:04:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.014+0000 7fdab6ffd640 20 client.4293.objecter get_session s=0x7fdabc041c90 osd=7 3 2023-11-25T05:04:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.014+0000 7fdab6ffd640 15 client.4293.objecter _session_command_op_assign 7 1 2023-11-25T05:04:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.014+0000 7fdab6ffd640 20 client.4293.objecter put_session s=0x7fdabc041c90 osd=7 4 2023-11-25T05:04:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.014+0000 7fdab6ffd640 10 client.4293.objecter _send_command 1 2023-11-25T05:04:36.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.014+0000 7fdab6ffd640 1 -- 172.21.15.7:0/2141132124 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdaa4075590 con 0x7fdaa4072ac0 2023-11-25T05:04:36.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.014+0000 7fdab6ffd640 20 client.4293.objecter dump_active .. 0 homeless 2023-11-25T05:04:36.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.014+0000 7fdac6a8f640 1 --2- 172.21.15.7:0/2141132124 >> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] conn(0x7fdaa4072ac0 0x7fdaa4074ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.014+0000 7fdab6ffd640 1 -- 172.21.15.7:0/2141132124 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdabc07b3a0 con 0x7fdac006f570 2023-11-25T05:04:36.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7fdab6ffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-11-25T05:04:36.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7fdac6a8f640 10 monclient: get_auth_request con 0x7fdaa4072ac0 auth_method 0 2023-11-25T05:04:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dd2ffd640 1 -- 172.21.15.7:0/2515178051 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8031830 con 0x7f9dd4149dc0 2023-11-25T05:04:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dd2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dd2ffd640 10 monclient: dump: 2023-11-25T05:04:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.030 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dd2ffd640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dd2ffd640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dd2ffd640 1 -- 172.21.15.7:0/2515178051 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc8031990 con 0x7f9dd4149dc0 2023-11-25T05:04:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dda6c2640 5 monclient: authenticate success, global_id 4303 2023-11-25T05:04:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dda6c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dd2ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dda6c2640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dda6c2640 4 set_mon_vals no callback set 2023-11-25T05:04:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dd2ffd640 1 -- 172.21.15.7:0/2515178051 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8031d20 con 0x7f9dd4149dc0 2023-11-25T05:04:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.015+0000 7f9dd2ffd640 10 monclient: dump: 2023-11-25T05:04:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.032 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.016+0000 7f9dda6c2640 10 monclient: shutdown 2023-11-25T05:04:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.016+0000 7f9dda6c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.016+0000 7f9dda6c2640 1 -- 172.21.15.7:0/2515178051 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd4149dc0 msgr2=0x7f9dd41509f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.016+0000 7f9dda6c2640 1 --2- 172.21.15.7:0/2515178051 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd4149dc0 0x7f9dd41509f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9dd41460e0 tx=0x7f9dc802b8d0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.016+0000 7fdac5a8d640 1 --2- 172.21.15.7:0/2141132124 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdaa406c850 0x7fdaa406ed10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdac013faa0 tx=0x7fdab0007990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.016+0000 7f9dda6c2640 1 -- 172.21.15.7:0/2515178051 shutdown_connections 2023-11-25T05:04:36.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.016+0000 7f9dda6c2640 1 --2- 172.21.15.7:0/2515178051 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd4149dc0 0x7f9dd41509f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.016+0000 7f9dda6c2640 1 --2- 172.21.15.7:0/2515178051 >> v2:172.21.15.7:3300/0 conn(0x7f9dd4143920 0x7f9dd4149880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.016+0000 7f9dda6c2640 1 -- 172.21.15.7:0/2515178051 >> 172.21.15.7:0/2515178051 conn(0x7f9dd40b01e0 msgr2=0x7f9dd40b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.016+0000 7fdab6ffd640 10 client.4293.objecter ms_handle_connect 0x7fdaa406c850 2023-11-25T05:04:36.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.016+0000 7f9dda6c2640 1 -- 172.21.15.7:0/2515178051 shutdown_connections 2023-11-25T05:04:36.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.019+0000 7fdac6a8f640 1 --2- 172.21.15.7:0/2141132124 >> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] conn(0x7fdaa4072ac0 0x7fdaa4074ee0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.019+0000 7fdab6ffd640 10 client.4293.objecter ms_handle_connect 0x7fdaa4072ac0 2023-11-25T05:04:36.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.016+0000 7f9dda6c2640 1 -- 172.21.15.7:0/2515178051 wait complete. 2023-11-25T05:04:36.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7f9dda6c2640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7f9dda6c2640 10 monclient: monmap: 2023-11-25T05:04:36.035 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.035 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:36.035 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:36.035 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:36.036 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.036 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.036 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:36.036 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:36.036 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:36.036 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa21bfff640 1 -- 172.21.15.7:0/928535431 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa220031830 con 0x7fa2240a8830 2023-11-25T05:04:36.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa21bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa21bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:36.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa21bfff640 10 monclient: dump: 2023-11-25T05:04:36.037 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.037 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.037 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.037 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.037 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.037 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.037 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.038 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.038 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.038 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa21bfff640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa21bfff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa21bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa22a36a640 5 monclient: authenticate success, global_id 4309 2023-11-25T05:04:36.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa22a36a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:36.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa21bfff640 1 -- 172.21.15.7:0/928535431 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa220031990 con 0x7fa2240a8830 2023-11-25T05:04:36.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa21bfff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa21bfff640 1 -- 172.21.15.7:0/928535431 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa220031d20 con 0x7fa2240a8830 2023-11-25T05:04:36.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa21bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa21bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:36.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa21bfff640 10 monclient: dump: 2023-11-25T05:04:36.039 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.039 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.039 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.039 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.039 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.039 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.039 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.039 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.040 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.040 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa22a36a640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:36.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.017+0000 7fa22a36a640 4 set_mon_vals no callback set 2023-11-25T05:04:36.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.018+0000 7fa22a36a640 10 monclient: shutdown 2023-11-25T05:04:36.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.018+0000 7fa22a36a640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.018+0000 7fa22a36a640 1 -- 172.21.15.7:0/928535431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2240a8830 msgr2=0x7fa2241423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.018+0000 7fa22a36a640 1 --2- 172.21.15.7:0/928535431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2240a8830 0x7fa2241423e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa2200097a0 tx=0x7fa22002b8d0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.019+0000 7fa22a36a640 1 -- 172.21.15.7:0/928535431 shutdown_connections 2023-11-25T05:04:36.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.019+0000 7fa22a36a640 1 --2- 172.21.15.7:0/928535431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2240a8830 0x7fa2241423e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.019+0000 7fa22a36a640 1 -- 172.21.15.7:0/928535431 >> 172.21.15.7:0/928535431 conn(0x7fa2240618d0 msgr2=0x7fa2240b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.019+0000 7fa22a36a640 1 -- 172.21.15.7:0/928535431 shutdown_connections 2023-11-25T05:04:36.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.019+0000 7fa22a36a640 1 -- 172.21.15.7:0/928535431 wait complete. 2023-11-25T05:04:36.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.018+0000 7f9dda6c2640 1 librados: starting msgr at 2023-11-25T05:04:36.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.020+0000 7fa22a36a640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.020+0000 7fa22a36a640 10 monclient: monmap: 2023-11-25T05:04:36.041 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.042 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:36.042 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:36.042 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:36.042 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.042 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.042 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:36.042 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:36.042 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:36.042 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.020+0000 7fa22a36a640 1 librados: starting msgr at 2023-11-25T05:04:36.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.020+0000 7fa22a36a640 1 librados: starting objecter 2023-11-25T05:04:36.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.020+0000 7fa22a36a640 1 Processor -- start 2023-11-25T05:04:36.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.018+0000 7f9dda6c2640 1 librados: starting objecter 2023-11-25T05:04:36.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.018+0000 7f9dda6c2640 1 Processor -- start 2023-11-25T05:04:36.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.020+0000 7f9dda6c2640 1 -- start start 2023-11-25T05:04:36.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.020+0000 7f9dda6c2640 1 librados: setting wanted keys 2023-11-25T05:04:36.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.020+0000 7f9dda6c2640 1 librados: calling monclient init 2023-11-25T05:04:36.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.020+0000 7f9dda6c2640 10 monclient: init 2023-11-25T05:04:36.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-25T05:04:36.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dd40af360 0x7f9dd412fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 10 monclient(hunting): picked mon.noname-a con 0x7f9dd40af360 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:36.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.020+0000 7fa22a36a640 1 -- start start 2023-11-25T05:04:36.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.020+0000 7fa22a36a640 1 librados: setting wanted keys 2023-11-25T05:04:36.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.020+0000 7fa22a36a640 1 librados: calling monclient init 2023-11-25T05:04:36.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.020+0000 7fa22a36a640 10 monclient: init 2023-11-25T05:04:36.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7fa22a36a640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7fa22a36a640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-25T05:04:36.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7fa22a36a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2240a8830 0x7fa224153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7fa22a36a640 10 monclient(hunting): picked mon.noname-b con 0x7fa2240a8830 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:36.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7fa22a36a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2240a8c10 0x7fa22413b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7fa22a36a640 10 monclient(hunting): picked mon.noname-a con 0x7fa2240a8c10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:36.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7fa22a36a640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa2240b2110 0x7fa2241e39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd4143920 0x7f9dd40b2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 10 monclient(hunting): picked mon.noname-b con 0x7f9dd4143920 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:36.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f5f91ffb640 1 -- 172.21.15.7:0/3037019212 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8402f800 con 0x7f5f94142bf0 2023-11-25T05:04:36.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f5f91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f5f91ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:36.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f5f91ffb640 10 monclient: dump: 2023-11-25T05:04:36.047 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.047 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.047 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.047 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.047 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.047 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.047 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.047 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.047 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.047 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f5f91ffb640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f5f91ffb640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f5f91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f5f91ffb640 1 -- 172.21.15.7:0/3037019212 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8402f960 con 0x7f5f94142bf0 2023-11-25T05:04:36.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f5f91ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f5f91ffb640 1 -- 172.21.15.7:0/3037019212 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8402fcf0 con 0x7f5f94142bf0 2023-11-25T05:04:36.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f5f91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f5f91ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:36.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f5f91ffb640 10 monclient: dump: 2023-11-25T05:04:36.048 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.049 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.049 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.049 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.049 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.049 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.049 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.049 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.049 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.049 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7f5f995b0640 5 monclient: authenticate success, global_id 4315 2023-11-25T05:04:36.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7f5f995b0640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:36.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7f5f995b0640 4 set_mon_vals no callback set 2023-11-25T05:04:36.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7fa22a36a640 10 monclient(hunting): picked mon.noname-c con 0x7fa2240b2110 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7fa22a36a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7fa22a36a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22406d800 con 0x7fa2240a8c10 2023-11-25T05:04:36.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7fa22a36a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7fa22a36a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22406dae0 con 0x7fa2240a8830 2023-11-25T05:04:36.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7fa22a36a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7fa22a36a640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa224154380 con 0x7fa2240b2110 2023-11-25T05:04:36.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7fa22a36a640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7fa22a36a640 10 monclient(hunting): authenticate will time out at 1402.264940s 2023-11-25T05:04:36.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9dd4149dc0 0x7f9dd4147d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 10 monclient(hunting): picked mon.noname-c con 0x7f9dd4149dc0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd412d710 con 0x7f9dd40af360 2023-11-25T05:04:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40a9770 con 0x7f9dd4143920 2023-11-25T05:04:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40b1740 con 0x7f9dd4149dc0 2023-11-25T05:04:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.021+0000 7f9dda6c2640 10 monclient(hunting): authenticate will time out at 1402.264503s 2023-11-25T05:04:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7f5f995b0640 10 monclient: shutdown 2023-11-25T05:04:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7f5f995b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7f5f995b0640 1 -- 172.21.15.7:0/3037019212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f94142bf0 msgr2=0x7f5f9413ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.022+0000 7f5f995b0640 1 --2- 172.21.15.7:0/3037019212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f94142bf0 0x7f5f9413ecf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5f840097a0 tx=0x7f5f8402b9e0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f5f995b0640 1 -- 172.21.15.7:0/3037019212 shutdown_connections 2023-11-25T05:04:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f5f995b0640 1 --2- 172.21.15.7:0/3037019212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f94142bf0 0x7f5f9413ecf0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f5f995b0640 1 -- 172.21.15.7:0/3037019212 >> 172.21.15.7:0/3037019212 conn(0x7f5f94061a30 msgr2=0x7f5f940b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f5f995b0640 1 -- 172.21.15.7:0/3037019212 shutdown_connections 2023-11-25T05:04:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f464cff9640 1 -- 172.21.15.7:0/1972786767 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4644032840 con 0x7f465014a360 2023-11-25T05:04:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f464cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f464cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f464cff9640 10 monclient: dump: 2023-11-25T05:04:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.055 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.055 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.055 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.055 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.055 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.055 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.055 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.055 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.055 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f464cff9640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f464cff9640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f464cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f464cff9640 1 -- 172.21.15.7:0/1972786767 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46440329c0 con 0x7f465014a360 2023-11-25T05:04:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f465694e640 5 monclient: authenticate success, global_id 4321 2023-11-25T05:04:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f465694e640 10 monclient: _renew_subs 2023-11-25T05:04:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f465694e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f465694e640 1 -- 172.21.15.7:0/1972786767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46501dbed0 con 0x7f465014a360 2023-11-25T05:04:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f465694e640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f465694e640 10 monclient: _renew_subs 2023-11-25T05:04:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f465694e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f465694e640 1 -- 172.21.15.7:0/1972786767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f465013ced0 con 0x7f465014a360 2023-11-25T05:04:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f465694e640 1 librados: init done 2023-11-25T05:04:36.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.023+0000 7f5f995b0640 1 -- 172.21.15.7:0/3037019212 wait complete. 2023-11-25T05:04:36.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f2068ff9640 1 -- 172.21.15.7:0/582799257 <== osd.0 v2:172.21.15.7:6824/2271997805 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f20580756d0 con 0x7f2058072c00 2023-11-25T05:04:36.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f2068ff9640 10 client.4287.objecter ms_dispatch 0x7f206c000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:04:36.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f2068ff9640 10 client.4287.objecter _finish_command 1 = system:0 2023-11-25T05:04:36.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f2068ff9640 20 client.4287.objecter put_session s=0x7f2058072530 osd=0 4 2023-11-25T05:04:36.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f2068ff9640 15 client.4287.objecter _session_command_op_remove 0 1 2023-11-25T05:04:36.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f9dd8c5f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9dd4149dc0 0x7f9dd4147d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f9dd8c5f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9dd4149dc0 0x7f9dd4147d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33664/0 (socket says 172.21.15.7:33664) 2023-11-25T05:04:36.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f9dd8c5f640 1 -- 172.21.15.7:0/1503077748 learned_addr learned my addr 172.21.15.7:0/1503077748 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f9dd8c5f640 10 monclient(hunting): get_auth_request con 0x7f9dd4149dc0 auth_method 0 2023-11-25T05:04:36.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f9dd8c5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f9dd8c5f640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f9dd8c5f640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7fa229b69640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa2240b2110 0x7fa2241e39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7fa229b69640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa2240b2110 0x7fa2241e39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33676/0 (socket says 172.21.15.7:33676) 2023-11-25T05:04:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7fa229b69640 1 -- 172.21.15.7:0/1476158480 learned_addr learned my addr 172.21.15.7:0/1476158480 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7fa229b69640 10 monclient(hunting): get_auth_request con 0x7fa2240b2110 auth_method 0 2023-11-25T05:04:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7fa229b69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7fa229b69640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7fa229b69640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7fa229368640 1 --2- 172.21.15.7:0/1476158480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2240a8830 0x7fa224153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7fa229368640 10 monclient(hunting): get_auth_request con 0x7fa2240a8830 auth_method 0 2023-11-25T05:04:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7fa229368640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7fa229368640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7fa229368640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.024+0000 7f5f995b0640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.024+0000 7f5f995b0640 10 monclient: monmap: 2023-11-25T05:04:36.060 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.060 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:36.060 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:36.060 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:36.060 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.060 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.060 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:36.061 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.024+0000 7f5f995b0640 1 librados: starting msgr at 2023-11-25T05:04:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.024+0000 7f5f995b0640 1 librados: starting objecter 2023-11-25T05:04:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.024+0000 7f5f995b0640 1 Processor -- start 2023-11-25T05:04:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.024+0000 7f464cff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.024+0000 7f464cff9640 1 -- 172.21.15.7:0/1972786767 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4644032d70 con 0x7f465014a360 2023-11-25T05:04:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.024+0000 7f464cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.024+0000 7f464cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.024+0000 7f464cff9640 10 monclient: dump: 2023-11-25T05:04:36.062 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.062 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.062 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.062 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.062 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.062 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.062 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.062 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.062 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.062 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.024+0000 7f464e7fc640 4 set_mon_vals no callback set 2023-11-25T05:04:36.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f5f995b0640 1 -- start start 2023-11-25T05:04:36.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f9dd37fe640 1 --2- 172.21.15.7:0/1503077748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd4143920 0x7f9dd40b2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f9dd37fe640 10 monclient(hunting): get_auth_request con 0x7f9dd4143920 auth_method 0 2023-11-25T05:04:36.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f9dd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f9dd37fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f9dd37fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f9dd8c5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f9dd8c5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f9dd8c5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7fa229368640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7fa229368640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7fa229368640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f5f995b0640 1 librados: setting wanted keys 2023-11-25T05:04:36.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f5f995b0640 1 librados: calling monclient init 2023-11-25T05:04:36.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f5f995b0640 10 monclient: init 2023-11-25T05:04:36.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f5f995b0640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f5f995b0640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-25T05:04:36.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f5f995b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f940a8a30 0x7f5f940b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f5f995b0640 10 monclient(hunting): picked mon.noname-b con 0x7f5f940a8a30 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:36.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.025+0000 7f5f995b0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f940a8e10 0x7f5f94145da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f5f995b0640 10 monclient(hunting): picked mon.noname-a con 0x7f5f940a8e10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:36.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f5f995b0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f94142bf0 0x7f5f9406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f5f995b0640 10 monclient(hunting): picked mon.noname-c con 0x7f5f94142bf0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f5f995b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f5f995b0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9406e110 con 0x7f5f940a8e10 2023-11-25T05:04:36.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f5f995b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f5f995b0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f94145610 con 0x7f5f940a8a30 2023-11-25T05:04:36.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f5f995b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f5f995b0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f941462e0 con 0x7f5f94142bf0 2023-11-25T05:04:36.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f5f995b0640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f5f995b0640 10 monclient(hunting): authenticate will time out at 1402.269555s 2023-11-25T05:04:36.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f465694e640 10 client.4321.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f465694e640 15 client.4321.objecter _session_command_op_assign -1 1 2023-11-25T05:04:36.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f465694e640 20 client.4321.objecter _get_session osd=-1 returning homeless 2023-11-25T05:04:36.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f465694e640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f465694e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.026+0000 7f465694e640 1 -- 172.21.15.7:0/1972786767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f46300028a0 con 0x7f465014a360 2023-11-25T05:04:36.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.028+0000 7f9dd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.028+0000 7f9dd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.028+0000 7f9dd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.031+0000 7fa229368640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2023-11-25T05:04:36.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.031+0000 7fa229368640 1 -- 172.21.15.7:0/1476158480 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa2240b2110 msgr2=0x7fa2241e39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.031+0000 7fa229368640 1 --2- 172.21.15.7:0/1476158480 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa2240b2110 0x7fa2241e39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.031+0000 7fa229368640 1 -- 172.21.15.7:0/1476158480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2240a8c10 msgr2=0x7fa22413b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.031+0000 7fa229368640 1 --2- 172.21.15.7:0/1476158480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2240a8c10 0x7fa22413b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.031+0000 7fa229368640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.031+0000 7fa229368640 1 monclient: found mon.noname-b 2023-11-25T05:04:36.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.031+0000 7fa229368640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.031+0000 7fa229368640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.031+0000 7fa229368640 1 -- 172.21.15.7:0/1476158480 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa220009400 con 0x7fa2240a8830 2023-11-25T05:04:36.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.033+0000 7f9dd37fe640 10 monclient(hunting): handle_auth_done global_id 4327 payload 824 2023-11-25T05:04:36.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.033+0000 7f9dd37fe640 1 -- 172.21.15.7:0/1503077748 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9dd4149dc0 msgr2=0x7f9dd4147d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.033+0000 7f9dd37fe640 1 --2- 172.21.15.7:0/1503077748 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9dd4149dc0 0x7f9dd4147d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.033+0000 7f9dd37fe640 1 -- 172.21.15.7:0/1503077748 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dd40af360 msgr2=0x7f9dd412fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.033+0000 7f9dd37fe640 1 --2- 172.21.15.7:0/1503077748 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dd40af360 0x7f9dd412fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.033+0000 7f9dd37fe640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.033+0000 7f9dd37fe640 1 monclient: found mon.noname-b 2023-11-25T05:04:36.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.033+0000 7f9dd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.033+0000 7f9dd37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.033+0000 7f9dd37fe640 1 -- 172.21.15.7:0/1503077748 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc8009400 con 0x7f9dd4143920 2023-11-25T05:04:36.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.036+0000 7feb98426640 10 client.4250.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-25T05:04:36.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.036+0000 7feb98426640 15 client.4250.objecter _session_command_op_assign -1 2 2023-11-25T05:04:36.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.036+0000 7feb98426640 20 client.4250.objecter _get_session s=0x7feb74072530 osd=3 3 2023-11-25T05:04:36.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.036+0000 7feb98426640 20 client.4250.objecter put_session s=0x7feb74072530 osd=3 3 2023-11-25T05:04:36.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.036+0000 7feb98426640 20 client.4250.objecter _get_session s=0x7feb74072530 osd=3 3 2023-11-25T05:04:36.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.036+0000 7feb98426640 15 client.4250.objecter _session_command_op_remove -1 2 2023-11-25T05:04:36.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.036+0000 7feb98426640 20 client.4250.objecter get_session s=0x7feb74072530 osd=3 3 2023-11-25T05:04:36.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.036+0000 7feb98426640 15 client.4250.objecter _session_command_op_assign 3 2 2023-11-25T05:04:36.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.036+0000 7feb98426640 20 client.4250.objecter put_session s=0x7feb74072530 osd=3 4 2023-11-25T05:04:36.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.036+0000 7feb98426640 10 client.4250.objecter _send_command 2 2023-11-25T05:04:36.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.036+0000 7feb98426640 1 -- 172.21.15.7:0/400856470 --> [v2:172.21.15.7:6816/59754388,v1:172.21.15.7:6817/59754388] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feb901cbfe0 con 0x7feb74072c00 2023-11-25T05:04:36.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.040+0000 7f5f937fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f94142bf0 0x7f5f9406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.041+0000 7f5f927fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f940a8e10 0x7f5f94145da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.042+0000 7f5f937fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f94142bf0 0x7f5f9406ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33678/0 (socket says 172.21.15.7:33678) 2023-11-25T05:04:36.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.042+0000 7f5f937fe640 1 -- 172.21.15.7:0/1898606146 learned_addr learned my addr 172.21.15.7:0/1898606146 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.042+0000 7f5f937fe640 10 monclient(hunting): get_auth_request con 0x7f5f94142bf0 auth_method 0 2023-11-25T05:04:36.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.042+0000 7f5f937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.042+0000 7f5f937fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.042+0000 7f5f937fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.043+0000 7f5f927fc640 10 monclient(hunting): get_auth_request con 0x7f5f940a8e10 auth_method 0 2023-11-25T05:04:36.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.043+0000 7f5f927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.043+0000 7f5f927fc640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.043+0000 7f5f927fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.043+0000 7f5f927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.043+0000 7f5f927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.043+0000 7f5f927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.046+0000 7f5f937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.046+0000 7f5f937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.046+0000 7f5f937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.046+0000 7f5f927fc640 10 monclient(hunting): handle_auth_done global_id 4296 payload 824 2023-11-25T05:04:36.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.046+0000 7f5f927fc640 1 -- 172.21.15.7:0/1898606146 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f94142bf0 msgr2=0x7f5f9406ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.046+0000 7f5f927fc640 1 --2- 172.21.15.7:0/1898606146 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f94142bf0 0x7f5f9406ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.046+0000 7f5f927fc640 1 -- 172.21.15.7:0/1898606146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f940a8a30 msgr2=0x7f5f940b69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.046+0000 7f5f927fc640 1 --2- 172.21.15.7:0/1898606146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f940a8a30 0x7f5f940b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.046+0000 7f5f927fc640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.046+0000 7f5f927fc640 1 monclient: found mon.noname-a 2023-11-25T05:04:36.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.046+0000 7f5f927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.046+0000 7f5f927fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.046+0000 7f5f927fc640 1 -- 172.21.15.7:0/1898606146 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f84009400 con 0x7f5f940a8e10 2023-11-25T05:04:36.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.048+0000 7feb86ffd640 1 -- 172.21.15.7:0/400856470 <== osd.3 v2:172.21.15.7:6816/59754388 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7feb901cbfe0 con 0x7feb74072c00 2023-11-25T05:04:36.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.048+0000 7feb86ffd640 10 client.4250.objecter ms_dispatch 0x7feb90000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:04:36.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.048+0000 7feb86ffd640 10 client.4250.objecter _finish_command 2 = system:0 2023-11-25T05:04:36.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.048+0000 7feb86ffd640 20 client.4250.objecter put_session s=0x7feb74072530 osd=3 4 2023-11-25T05:04:36.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.048+0000 7feb86ffd640 15 client.4250.objecter _session_command_op_remove 3 2 2023-11-25T05:04:36.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 10 librados: watch_flush enter 2023-11-25T05:04:36.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 10 librados: watch_flush exit 2023-11-25T05:04:36.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 10 client.4250.objecter close_session for osd.3 2023-11-25T05:04:36.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 1 -- 172.21.15.7:0/400856470 >> [v2:172.21.15.7:6816/59754388,v1:172.21.15.7:6817/59754388] conn(0x7feb74072c00 msgr2=0x7feb74075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 1 --2- 172.21.15.7:0/400856470 >> [v2:172.21.15.7:6816/59754388,v1:172.21.15.7:6817/59754388] conn(0x7feb74072c00 0x7feb74075020 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 20 client.4250.objecter put_session s=0x7feb74072530 osd=3 1 2023-11-25T05:04:36.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 20 client.4250.objecter shutdown clearing up homeless session... 2023-11-25T05:04:36.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 10 client.4250.objecter successfully canceled tick 2023-11-25T05:04:36.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 1 -- 172.21.15.7:0/400856470 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7feb7406c820 msgr2=0x7feb7406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 1 --2- 172.21.15.7:0/400856470 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7feb7406c820 0x7feb7406ece0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feb88007fd0 tx=0x7feb88007d40 comp rx=0 tx=0).stop 2023-11-25T05:04:36.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 10 monclient: shutdown 2023-11-25T05:04:36.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 1 -- 172.21.15.7:0/400856470 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb90070220 msgr2=0x7feb901cb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 1 --2- 172.21.15.7:0/400856470 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb90070220 0x7feb901cb380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009870 tx=0x7feb8c0046c0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb969c3640 1 -- 172.21.15.7:0/400856470 reap_dead start 2023-11-25T05:04:36.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 1 -- 172.21.15.7:0/400856470 shutdown_connections 2023-11-25T05:04:36.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 1 -- 172.21.15.7:0/400856470 >> 172.21.15.7:0/400856470 conn(0x7feb9006a6e0 msgr2=0x7feb901c9930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 1 -- 172.21.15.7:0/400856470 shutdown_connections 2023-11-25T05:04:36.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 1 -- 172.21.15.7:0/400856470 wait complete. 2023-11-25T05:04:36.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.049+0000 7feb98426640 1 librados: shutdown 2023-11-25T05:04:36.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.050+0000 7f20729ce640 10 client.4287.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-25T05:04:36.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.050+0000 7f20729ce640 15 client.4287.objecter _session_command_op_assign -1 2 2023-11-25T05:04:36.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.050+0000 7f20729ce640 20 client.4287.objecter _get_session s=0x7f2058072530 osd=0 3 2023-11-25T05:04:36.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.050+0000 7f20729ce640 20 client.4287.objecter put_session s=0x7f2058072530 osd=0 3 2023-11-25T05:04:36.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.050+0000 7f20729ce640 20 client.4287.objecter _get_session s=0x7f2058072530 osd=0 3 2023-11-25T05:04:36.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.050+0000 7f20729ce640 15 client.4287.objecter _session_command_op_remove -1 2 2023-11-25T05:04:36.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.050+0000 7f20729ce640 20 client.4287.objecter get_session s=0x7f2058072530 osd=0 3 2023-11-25T05:04:36.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.050+0000 7f20729ce640 15 client.4287.objecter _session_command_op_assign 0 2 2023-11-25T05:04:36.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.050+0000 7f20729ce640 20 client.4287.objecter put_session s=0x7f2058072530 osd=0 4 2023-11-25T05:04:36.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.050+0000 7f20729ce640 10 client.4287.objecter _send_command 2 2023-11-25T05:04:36.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.050+0000 7f20729ce640 1 -- 172.21.15.7:0/582799257 --> [v2:172.21.15.7:6824/2271997805,v1:172.21.15.7:6825/2271997805] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f206c13c400 con 0x7f2058072c00 2023-11-25T05:04:36.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.051+0000 7f5f927fc640 1 --2- 172.21.15.7:0/1898606146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f940a8e10 0x7f5f94145da0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5f940b8940 tx=0x7f5f84004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.051+0000 7f5f7b7fe640 10 client.?.objecter ms_handle_connect 0x7f5f940a8e10 2023-11-25T05:04:36.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.051+0000 7f5f7b7fe640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:36.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7fa229368640 1 --2- 172.21.15.7:0/1476158480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2240a8830 0x7fa224153c30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa220033000 tx=0x7fa2200046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7fa219ffb640 10 client.?.objecter ms_handle_connect 0x7fa2240a8830 2023-11-25T05:04:36.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7fa219ffb640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:36.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f7b7fe640 1 -- 172.21.15.7:0/1898606146 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8403c040 con 0x7f5f940a8e10 2023-11-25T05:04:36.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f7b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:36.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f7b7fe640 10 monclient: dump: 2023-11-25T05:04:36.080 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.080 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.081 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.081 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.081 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.081 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.081 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.081 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.081 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.081 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f7b7fe640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f7b7fe640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f7b7fe640 1 -- 172.21.15.7:0/1898606146 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f84037d90 con 0x7f5f940a8e10 2023-11-25T05:04:36.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f7b7fe640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f7b7fe640 1 -- 172.21.15.7:0/1898606146 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f84041680 con 0x7f5f940a8e10 2023-11-25T05:04:36.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f7b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:36.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f7b7fe640 10 monclient: dump: 2023-11-25T05:04:36.082 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.082 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.082 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.082 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.082 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.082 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.082 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.082 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.083 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.083 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f917fa640 4 set_mon_vals no callback set 2023-11-25T05:04:36.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f995b0640 5 monclient: authenticate success, global_id 4296 2023-11-25T05:04:36.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f995b0640 10 monclient: _renew_subs 2023-11-25T05:04:36.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f995b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f995b0640 1 -- 172.21.15.7:0/1898606146 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9406e380 con 0x7f5f940a8e10 2023-11-25T05:04:36.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f995b0640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f995b0640 10 monclient: _renew_subs 2023-11-25T05:04:36.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f995b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f995b0640 1 -- 172.21.15.7:0/1898606146 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f94145880 con 0x7f5f940a8e10 2023-11-25T05:04:36.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f5f995b0640 1 librados: init done 2023-11-25T05:04:36.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f464cff9640 1 -- 172.21.15.7:0/1972786767 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f464403b3f0 con 0x7f465014a360 2023-11-25T05:04:36.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.056+0000 7f464cff9640 1 --2- 172.21.15.7:0/1972786767 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f463c06c820 0x7f463c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.057+0000 7f464ffff640 1 --2- 172.21.15.7:0/1972786767 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f463c06c820 0x7f463c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.057+0000 7f464ffff640 10 monclient: get_auth_request con 0x7f463c06c820 auth_method 0 2023-11-25T05:04:36.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.057+0000 7f464ffff640 1 --2- 172.21.15.7:0/1972786767 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f463c06c820 0x7f463c06ece0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4640000f60 tx=0x7f46400079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.057+0000 7f464cff9640 10 client.4321.objecter ms_handle_connect 0x7f463c06c820 2023-11-25T05:04:36.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.058+0000 7f5f995b0640 10 client.4296.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.058+0000 7f5f995b0640 15 client.4296.objecter _session_command_op_assign -1 1 2023-11-25T05:04:36.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.058+0000 7f5f995b0640 20 client.4296.objecter _get_session osd=-1 returning homeless 2023-11-25T05:04:36.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.058+0000 7f5f995b0640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.058+0000 7f5f995b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.058+0000 7f5f995b0640 1 -- 172.21.15.7:0/1898606146 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5f9413c680 con 0x7f5f940a8e10 2023-11-25T05:04:36.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.060+0000 7f2068ff9640 1 -- 172.21.15.7:0/582799257 <== osd.0 v2:172.21.15.7:6824/2271997805 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f206c13c400 con 0x7f2058072c00 2023-11-25T05:04:36.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.060+0000 7f2068ff9640 10 client.4287.objecter ms_dispatch 0x7f206c000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:04:36.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.060+0000 7f2068ff9640 10 client.4287.objecter _finish_command 2 = system:0 2023-11-25T05:04:36.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.060+0000 7f2068ff9640 20 client.4287.objecter put_session s=0x7f2058072530 osd=0 4 2023-11-25T05:04:36.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.060+0000 7f2068ff9640 15 client.4287.objecter _session_command_op_remove 0 2 2023-11-25T05:04:36.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f20729ce640 10 librados: watch_flush enter 2023-11-25T05:04:36.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 1 -- 172.21.15.7:0/1972786767 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f464404b2c0 con 0x7f465014a360 2023-11-25T05:04:36.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 10 client.4321.objecter ms_dispatch 0x7f4650000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:36.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 3 client.4321.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:36.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 3 client.4321.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:36.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f20729ce640 10 librados: watch_flush exit 2023-11-25T05:04:36.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f20729ce640 10 client.4287.objecter close_session for osd.0 2023-11-25T05:04:36.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f20729ce640 1 -- 172.21.15.7:0/582799257 >> [v2:172.21.15.7:6824/2271997805,v1:172.21.15.7:6825/2271997805] conn(0x7f2058072c00 msgr2=0x7f2058075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f20729ce640 1 --2- 172.21.15.7:0/582799257 >> [v2:172.21.15.7:6824/2271997805,v1:172.21.15.7:6825/2271997805] conn(0x7f2058072c00 0x7f2058075020 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f20729ce640 20 client.4287.objecter put_session s=0x7f2058072530 osd=0 1 2023-11-25T05:04:36.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f20729ce640 20 client.4287.objecter shutdown clearing up homeless session... 2023-11-25T05:04:36.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f20729ce640 10 client.4287.objecter successfully canceled tick 2023-11-25T05:04:36.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 10 client.4321.objecter checking command 1 2023-11-25T05:04:36.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 1 --2- 172.21.15.7:0/1972786767 >> [v2:172.21.15.90:6810/132027349,v1:172.21.15.90:6811/132027349] conn(0x7f463c072c00 0x7f463c075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 20 client.4321.objecter _get_session s=0x7f463c072530 osd=5 3 2023-11-25T05:04:36.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 20 client.4321.objecter put_session s=0x7f463c072530 osd=5 3 2023-11-25T05:04:36.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 15 client.4321.objecter _session_command_op_remove -1 1 2023-11-25T05:04:36.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 20 client.4321.objecter _get_session s=0x7f463c072530 osd=5 3 2023-11-25T05:04:36.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 20 client.4321.objecter get_session s=0x7f463c072530 osd=5 3 2023-11-25T05:04:36.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 15 client.4321.objecter _session_command_op_assign 5 1 2023-11-25T05:04:36.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 20 client.4321.objecter put_session s=0x7f463c072530 osd=5 4 2023-11-25T05:04:36.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 10 client.4321.objecter _send_command 1 2023-11-25T05:04:36.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 1 -- 172.21.15.7:0/1972786767 --> [v2:172.21.15.90:6810/132027349,v1:172.21.15.90:6811/132027349] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f463c0756d0 con 0x7f463c072c00 2023-11-25T05:04:36.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f464cff9640 20 client.4321.objecter dump_active .. 0 homeless 2023-11-25T05:04:36.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f20729ce640 1 -- 172.21.15.7:0/582799257 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f205806c820 msgr2=0x7f205806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f20729ce640 1 --2- 172.21.15.7:0/582799257 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f205806c820 0x7f205806ece0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f206400ab90 tx=0x7f2064009250 comp rx=0 tx=0).stop 2023-11-25T05:04:36.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f20729ce640 10 monclient: shutdown 2023-11-25T05:04:36.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f20729ce640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f20729ce640 1 -- 172.21.15.7:0/582799257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f206c142a20 msgr2=0x7f206c13b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f20729ce640 1 --2- 172.21.15.7:0/582799257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f206c142a20 0x7f206c13b880 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f205c034040 tx=0x7f205c0046c0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f2070f6b640 1 -- 172.21.15.7:0/582799257 reap_dead start 2023-11-25T05:04:36.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.061+0000 7f20729ce640 1 -- 172.21.15.7:0/582799257 shutdown_connections 2023-11-25T05:04:36.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.062+0000 7f20729ce640 1 -- 172.21.15.7:0/582799257 >> 172.21.15.7:0/582799257 conn(0x7f206c061a30 msgr2=0x7f206c0b2630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.062+0000 7f20729ce640 1 -- 172.21.15.7:0/582799257 shutdown_connections 2023-11-25T05:04:36.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.062+0000 7f20729ce640 1 -- 172.21.15.7:0/582799257 wait complete. 2023-11-25T05:04:36.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.062+0000 7f20729ce640 1 librados: shutdown 2023-11-25T05:04:36.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.062+0000 7f464f7fe640 1 --2- 172.21.15.7:0/1972786767 >> [v2:172.21.15.90:6810/132027349,v1:172.21.15.90:6811/132027349] conn(0x7f463c072c00 0x7f463c075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.097 INFO:teuthology.orchestra.run.smithi007.stdout:68719476741 2023-11-25T05:04:36.097 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-25T05:04:36.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.062+0000 7f464f7fe640 10 monclient: get_auth_request con 0x7f463c072c00 auth_method 0 2023-11-25T05:04:36.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.063+0000 7f464cff9640 1 -- 172.21.15.7:0/1972786767 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4644076740 con 0x7f465014a360 2023-11-25T05:04:36.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.063+0000 7f464cff9640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-11-25T05:04:36.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.063+0000 7f9dd37fe640 1 --2- 172.21.15.7:0/1503077748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd4143920 0x7f9dd40b2e20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9dc400e850 tx=0x7f9dc400ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.063+0000 7f9dd0ff9640 10 client.?.objecter ms_handle_connect 0x7f9dd4143920 2023-11-25T05:04:36.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.063+0000 7f9dd0ff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:36.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.065+0000 7f464f7fe640 1 --2- 172.21.15.7:0/1972786767 >> [v2:172.21.15.90:6810/132027349,v1:172.21.15.90:6811/132027349] conn(0x7f463c072c00 0x7f463c075020 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.065+0000 7f464cff9640 10 client.4321.objecter ms_handle_connect 0x7f463c072c00 2023-11-25T05:04:36.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.067+0000 7fdab6ffd640 1 -- 172.21.15.7:0/2141132124 <== osd.7 v2:172.21.15.90:6802/815268397 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fdaa4075590 con 0x7fdaa4072ac0 2023-11-25T05:04:36.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.067+0000 7fdab6ffd640 10 client.4293.objecter ms_dispatch 0x7fdac0000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:04:36.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.067+0000 7fdab6ffd640 10 client.4293.objecter _finish_command 1 = system:0 2023-11-25T05:04:36.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.067+0000 7fdab6ffd640 20 client.4293.objecter put_session s=0x7fdabc041c90 osd=7 4 2023-11-25T05:04:36.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.067+0000 7fdab6ffd640 15 client.4293.objecter _session_command_op_remove 7 1 2023-11-25T05:04:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.072+0000 7fa219ffb640 1 -- 172.21.15.7:0/1476158480 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22003ba70 con 0x7fa2240a8830 2023-11-25T05:04:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.072+0000 7fa219ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.072+0000 7fa219ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.072+0000 7fa219ffb640 10 monclient: dump: 2023-11-25T05:04:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.102 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.072+0000 7fa219ffb640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.072+0000 7fa219ffb640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.072+0000 7fa219ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.072+0000 7fa22a36a640 5 monclient: authenticate success, global_id 4324 2023-11-25T05:04:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.072+0000 7fa22a36a640 10 monclient: _renew_subs 2023-11-25T05:04:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.072+0000 7fa22a36a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.072+0000 7fa22a36a640 1 -- 172.21.15.7:0/1476158480 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22406f0d0 con 0x7fa2240a8830 2023-11-25T05:04:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.073+0000 7fa22a36a640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.073+0000 7fa22a36a640 10 monclient: _renew_subs 2023-11-25T05:04:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.073+0000 7fa22a36a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.073+0000 7fa22a36a640 1 -- 172.21.15.7:0/1476158480 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22406e150 con 0x7fa2240a8830 2023-11-25T05:04:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.073+0000 7fa22a36a640 1 librados: init done 2023-11-25T05:04:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.074+0000 7fa22a36a640 10 client.4324.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.074+0000 7fa22a36a640 15 client.4324.objecter _session_command_op_assign -1 1 2023-11-25T05:04:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.074+0000 7fa22a36a640 20 client.4324.objecter _get_session osd=-1 returning homeless 2023-11-25T05:04:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.074+0000 7fa22a36a640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.074+0000 7fa22a36a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.074+0000 7fa22a36a640 1 -- 172.21.15.7:0/1476158480 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa22406dc20 con 0x7fa2240a8830 2023-11-25T05:04:36.104 INFO:teuthology.orchestra.run.smithi007.stdout:68719476741 2023-11-25T05:04:36.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-25T05:04:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.078+0000 7f9dd0ff9640 1 -- 172.21.15.7:0/1503077748 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc40095b0 con 0x7f9dd4143920 2023-11-25T05:04:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.078+0000 7f9dd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.078+0000 7f9dd0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.078+0000 7f9dd0ff9640 10 monclient: dump: 2023-11-25T05:04:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.106 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.078+0000 7f9dd0ff9640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.078+0000 7f9dd0ff9640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.078+0000 7f9dd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.078+0000 7f9dd0ff9640 1 -- 172.21.15.7:0/1503077748 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc400c940 con 0x7f9dd4143920 2023-11-25T05:04:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.078+0000 7f9dd0ff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.078+0000 7f9dd0ff9640 1 -- 172.21.15.7:0/1503077748 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc400ccd0 con 0x7f9dd4143920 2023-11-25T05:04:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.078+0000 7f9dd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.078+0000 7f9dd0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.078+0000 7f9dd0ff9640 10 monclient: dump: 2023-11-25T05:04:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.108 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.078+0000 7f9dd27fc640 4 set_mon_vals no callback set 2023-11-25T05:04:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7f9dda6c2640 5 monclient: authenticate success, global_id 4327 2023-11-25T05:04:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7f9dda6c2640 10 monclient: _renew_subs 2023-11-25T05:04:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7f9dda6c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7f9dda6c2640 1 -- 172.21.15.7:0/1503077748 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd41300a0 con 0x7f9dd4143920 2023-11-25T05:04:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7f9dda6c2640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7f9dda6c2640 10 monclient: _renew_subs 2023-11-25T05:04:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7f9dda6c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7f9dda6c2640 1 -- 172.21.15.7:0/1503077748 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd40ad1d0 con 0x7f9dd4143920 2023-11-25T05:04:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7f9dda6c2640 1 librados: init done 2023-11-25T05:04:36.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7fa219ffb640 1 -- 172.21.15.7:0/1476158480 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa22003bbd0 con 0x7fa2240a8830 2023-11-25T05:04:36.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7fa219ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7fa219ffb640 1 -- 172.21.15.7:0/1476158480 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22004e450 con 0x7fa2240a8830 2023-11-25T05:04:36.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7fa219ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7fa219ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:36.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7fa219ffb640 10 monclient: dump: 2023-11-25T05:04:36.109 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.109 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.109 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.109 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.110 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.110 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.110 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.110 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.110 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.110 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.079+0000 7fa21b7fe640 4 set_mon_vals no callback set 2023-11-25T05:04:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.080+0000 7f9dda6c2640 10 client.4327.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.080+0000 7f9dda6c2640 15 client.4327.objecter _session_command_op_assign -1 1 2023-11-25T05:04:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.080+0000 7f9dda6c2640 20 client.4327.objecter _get_session osd=-1 returning homeless 2023-11-25T05:04:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.080+0000 7f9dda6c2640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.080+0000 7f9dda6c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.080+0000 7f9dda6c2640 1 -- 172.21.15.7:0/1503077748 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9dd406fdc0 con 0x7f9dd4143920 2023-11-25T05:04:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.083+0000 7fdac84f2640 10 client.4293.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-25T05:04:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.083+0000 7fdac84f2640 15 client.4293.objecter _session_command_op_assign -1 2 2023-11-25T05:04:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.083+0000 7fdac84f2640 20 client.4293.objecter _get_session s=0x7fdabc041c90 osd=7 3 2023-11-25T05:04:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.083+0000 7fdac84f2640 20 client.4293.objecter put_session s=0x7fdabc041c90 osd=7 3 2023-11-25T05:04:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.083+0000 7fdac84f2640 20 client.4293.objecter _get_session s=0x7fdabc041c90 osd=7 3 2023-11-25T05:04:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.083+0000 7fdac84f2640 15 client.4293.objecter _session_command_op_remove -1 2 2023-11-25T05:04:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.083+0000 7fdac84f2640 20 client.4293.objecter get_session s=0x7fdabc041c90 osd=7 3 2023-11-25T05:04:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.083+0000 7fdac84f2640 15 client.4293.objecter _session_command_op_assign 7 2 2023-11-25T05:04:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.083+0000 7fdac84f2640 20 client.4293.objecter put_session s=0x7fdabc041c90 osd=7 4 2023-11-25T05:04:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.083+0000 7fdac84f2640 10 client.4293.objecter _send_command 2 2023-11-25T05:04:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.083+0000 7fdac84f2640 1 -- 172.21.15.7:0/2141132124 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdac013f690 con 0x7fdaa4072ac0 2023-11-25T05:04:36.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.084+0000 7f5f7b7fe640 1 -- 172.21.15.7:0/1898606146 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5f840417e0 con 0x7f5f940a8e10 2023-11-25T05:04:36.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.085+0000 7f5f7b7fe640 1 --2- 172.21.15.7:0/1898606146 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f5f8006c820 0x7f5f8006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.085+0000 7f5f92ffd640 1 --2- 172.21.15.7:0/1898606146 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f5f8006c820 0x7f5f8006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.085+0000 7f5f92ffd640 10 monclient: get_auth_request con 0x7f5f8006c820 auth_method 0 2023-11-25T05:04:36.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.085+0000 7f5f7b7fe640 10 client.4296.objecter ms_handle_connect 0x7f5f8006c820 2023-11-25T05:04:36.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.085+0000 7f5f92ffd640 1 --2- 172.21.15.7:0/1898606146 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f5f8006c820 0x7f5f8006ece0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5f88000f60 tx=0x7f5f880079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 1 -- 172.21.15.7:0/1898606146 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5f8407a740 con 0x7f5f940a8e10 2023-11-25T05:04:36.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 10 client.4296.objecter ms_dispatch 0x7f5f94000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:36.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 3 client.4296.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:36.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 3 client.4296.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:36.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 10 client.4296.objecter checking command 1 2023-11-25T05:04:36.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 1 --2- 172.21.15.7:0/1898606146 >> [v2:172.21.15.90:6820/1576115380,v1:172.21.15.90:6823/1576115380] conn(0x7f5f80072c00 0x7f5f80075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 20 client.4296.objecter _get_session s=0x7f5f80072530 osd=4 3 2023-11-25T05:04:36.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 20 client.4296.objecter put_session s=0x7f5f80072530 osd=4 3 2023-11-25T05:04:36.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 15 client.4296.objecter _session_command_op_remove -1 1 2023-11-25T05:04:36.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 20 client.4296.objecter _get_session s=0x7f5f80072530 osd=4 3 2023-11-25T05:04:36.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 20 client.4296.objecter get_session s=0x7f5f80072530 osd=4 3 2023-11-25T05:04:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 15 client.4296.objecter _session_command_op_assign 4 1 2023-11-25T05:04:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 20 client.4296.objecter put_session s=0x7f5f80072530 osd=4 4 2023-11-25T05:04:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 10 client.4296.objecter _send_command 1 2023-11-25T05:04:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 1 -- 172.21.15.7:0/1898606146 --> [v2:172.21.15.90:6820/1576115380,v1:172.21.15.90:6823/1576115380] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f800756d0 con 0x7f5f80072c00 2023-11-25T05:04:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 20 client.4296.objecter dump_active .. 0 homeless 2023-11-25T05:04:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 1 -- 172.21.15.7:0/1898606146 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f84076a80 con 0x7f5f940a8e10 2023-11-25T05:04:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.090+0000 7f5f7b7fe640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-11-25T05:04:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.091+0000 7f5f937fe640 1 --2- 172.21.15.7:0/1898606146 >> [v2:172.21.15.90:6820/1576115380,v1:172.21.15.90:6823/1576115380] conn(0x7f5f80072c00 0x7f5f80075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.091+0000 7f5f937fe640 10 monclient: get_auth_request con 0x7f5f80072c00 auth_method 0 2023-11-25T05:04:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.096+0000 7f5f937fe640 1 --2- 172.21.15.7:0/1898606146 >> [v2:172.21.15.90:6820/1576115380,v1:172.21.15.90:6823/1576115380] conn(0x7f5f80072c00 0x7f5f80075020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.096+0000 7f5f7b7fe640 10 client.4296.objecter ms_handle_connect 0x7f5f80072c00 2023-11-25T05:04:36.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.109+0000 7fdab6ffd640 1 -- 172.21.15.7:0/2141132124 <== osd.7 v2:172.21.15.90:6802/815268397 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdac013f690 con 0x7fdaa4072ac0 2023-11-25T05:04:36.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.109+0000 7fdab6ffd640 10 client.4293.objecter ms_dispatch 0x7fdac0000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:04:36.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.109+0000 7fdab6ffd640 10 client.4293.objecter _finish_command 2 = system:0 2023-11-25T05:04:36.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.109+0000 7fdab6ffd640 20 client.4293.objecter put_session s=0x7fdabc041c90 osd=7 4 2023-11-25T05:04:36.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.109+0000 7fdab6ffd640 15 client.4293.objecter _session_command_op_remove 7 2 2023-11-25T05:04:36.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 10 librados: watch_flush enter 2023-11-25T05:04:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 10 librados: watch_flush exit 2023-11-25T05:04:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 10 client.4293.objecter close_session for osd.7 2023-11-25T05:04:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 1 -- 172.21.15.7:0/2141132124 >> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] conn(0x7fdaa4072ac0 msgr2=0x7fdaa4074ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 1 --2- 172.21.15.7:0/2141132124 >> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] conn(0x7fdaa4072ac0 0x7fdaa4074ee0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 20 client.4293.objecter put_session s=0x7fdabc041c90 osd=7 1 2023-11-25T05:04:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 20 client.4293.objecter shutdown clearing up homeless session... 2023-11-25T05:04:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 10 client.4293.objecter successfully canceled tick 2023-11-25T05:04:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 1 -- 172.21.15.7:0/2141132124 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdaa406c850 msgr2=0x7fdaa406ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 1 --2- 172.21.15.7:0/2141132124 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdaa406c850 0x7fdaa406ed10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdac013faa0 tx=0x7fdab0007990 comp rx=0 tx=0).stop 2023-11-25T05:04:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 10 monclient: shutdown 2023-11-25T05:04:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 1 -- 172.21.15.7:0/2141132124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdac006f570 msgr2=0x7fdac01407f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 1 --2- 172.21.15.7:0/2141132124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdac006f570 0x7fdac01407f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdabc000c00 tx=0x7fdabc0046c0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac6a8f640 1 -- 172.21.15.7:0/2141132124 reap_dead start 2023-11-25T05:04:36.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 1 -- 172.21.15.7:0/2141132124 shutdown_connections 2023-11-25T05:04:36.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 1 -- 172.21.15.7:0/2141132124 >> 172.21.15.7:0/2141132124 conn(0x7fdac006c100 msgr2=0x7fdac0138e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 1 -- 172.21.15.7:0/2141132124 shutdown_connections 2023-11-25T05:04:36.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 1 -- 172.21.15.7:0/2141132124 wait complete. 2023-11-25T05:04:36.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.110+0000 7fdac84f2640 1 librados: shutdown 2023-11-25T05:04:36.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.117+0000 7fa219ffb640 1 -- 172.21.15.7:0/1476158480 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa22004e5b0 con 0x7fa2240a8830 2023-11-25T05:04:36.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.118+0000 7fa219ffb640 1 --2- 172.21.15.7:0/1476158480 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fa20806c800 0x7fa20806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.118+0000 7fa228b67640 1 --2- 172.21.15.7:0/1476158480 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fa20806c800 0x7fa20806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.118+0000 7fa228b67640 10 monclient: get_auth_request con 0x7fa20806c800 auth_method 0 2023-11-25T05:04:36.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.119+0000 7fa228b67640 1 --2- 172.21.15.7:0/1476158480 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fa20806c800 0x7fa20806ecc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa2241545c0 tx=0x7fa214007a40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.119+0000 7fa219ffb640 10 client.4324.objecter ms_handle_connect 0x7fa20806c800 2023-11-25T05:04:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 1 -- 172.21.15.7:0/1476158480 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa22007b2a0 con 0x7fa2240a8830 2023-11-25T05:04:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 10 client.4324.objecter ms_dispatch 0x7fa224000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 3 client.4324.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 3 client.4324.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7f464cff9640 1 -- 172.21.15.7:0/1972786767 <== osd.5 v2:172.21.15.90:6810/132027349 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f463c0756d0 con 0x7f463c072c00 2023-11-25T05:04:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7f464cff9640 10 client.4321.objecter ms_dispatch 0x7f4650000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:04:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7f464cff9640 10 client.4321.objecter _finish_command 1 = system:0 2023-11-25T05:04:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7f464cff9640 20 client.4321.objecter put_session s=0x7f463c072530 osd=5 4 2023-11-25T05:04:36.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7f464cff9640 15 client.4321.objecter _session_command_op_remove 5 1 2023-11-25T05:04:36.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 10 client.4324.objecter checking command 1 2023-11-25T05:04:36.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 1 --2- 172.21.15.7:0/1476158480 >> [v2:172.21.15.7:6808/1372486177,v1:172.21.15.7:6809/1372486177] conn(0x7fa208072be0 0x7fa208075000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 20 client.4324.objecter _get_session s=0x7fa208072510 osd=2 3 2023-11-25T05:04:36.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 20 client.4324.objecter put_session s=0x7fa208072510 osd=2 3 2023-11-25T05:04:36.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 15 client.4324.objecter _session_command_op_remove -1 1 2023-11-25T05:04:36.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 20 client.4324.objecter _get_session s=0x7fa208072510 osd=2 3 2023-11-25T05:04:36.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 20 client.4324.objecter get_session s=0x7fa208072510 osd=2 3 2023-11-25T05:04:36.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 15 client.4324.objecter _session_command_op_assign 2 1 2023-11-25T05:04:36.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 20 client.4324.objecter put_session s=0x7fa208072510 osd=2 4 2023-11-25T05:04:36.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 10 client.4324.objecter _send_command 1 2023-11-25T05:04:36.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 1 -- 172.21.15.7:0/1476158480 --> [v2:172.21.15.7:6808/1372486177,v1:172.21.15.7:6809/1372486177] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2080756b0 con 0x7fa208072be0 2023-11-25T05:04:36.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.120+0000 7fa219ffb640 20 client.4324.objecter dump_active .. 0 homeless 2023-11-25T05:04:36.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.121+0000 7fa229b69640 1 --2- 172.21.15.7:0/1476158480 >> [v2:172.21.15.7:6808/1372486177,v1:172.21.15.7:6809/1372486177] conn(0x7fa208072be0 0x7fa208075000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.121+0000 7fa229b69640 10 monclient: get_auth_request con 0x7fa208072be0 auth_method 0 2023-11-25T05:04:36.127 INFO:teuthology.orchestra.run.smithi007.stdout:55834574852 2023-11-25T05:04:36.128 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-25T05:04:36.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.126+0000 7fa219ffb640 10 client.4324.objecter ms_handle_connect 0x7fa208072be0 2023-11-25T05:04:36.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.126+0000 7fa229b69640 1 --2- 172.21.15.7:0/1476158480 >> [v2:172.21.15.7:6808/1372486177,v1:172.21.15.7:6809/1372486177] conn(0x7fa208072be0 0x7fa208075000 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.133+0000 7f9dd0ff9640 1 -- 172.21.15.7:0/1503077748 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9dc4013470 con 0x7f9dd4143920 2023-11-25T05:04:36.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.133+0000 7f9dd0ff9640 1 --2- 172.21.15.7:0/1503077748 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f9dc006c800 0x7f9dc006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.134+0000 7f9dd3fff640 1 --2- 172.21.15.7:0/1503077748 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f9dc006c800 0x7f9dc006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.134+0000 7f9dd3fff640 10 monclient: get_auth_request con 0x7f9dc006c800 auth_method 0 2023-11-25T05:04:36.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.134+0000 7f9dd0ff9640 10 client.4327.objecter ms_handle_connect 0x7f9dc006c800 2023-11-25T05:04:36.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.134+0000 7f9dd3fff640 1 --2- 172.21.15.7:0/1503077748 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f9dc006c800 0x7f9dc006ecc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc002070 tx=0x7f9dcc001660 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.134+0000 7f9dd0ff9640 1 -- 172.21.15.7:0/1503077748 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9dc400ce30 con 0x7f9dd4143920 2023-11-25T05:04:36.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.134+0000 7f9dd0ff9640 10 client.4327.objecter ms_dispatch 0x7f9dd4000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:36.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.134+0000 7f9dd0ff9640 3 client.4327.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:36.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.134+0000 7f9dd0ff9640 3 client.4327.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:36.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.135+0000 7f9dd0ff9640 10 client.4327.objecter checking command 1 2023-11-25T05:04:36.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.135+0000 7f9dd0ff9640 1 --2- 172.21.15.7:0/1503077748 >> [v2:172.21.15.7:6800/3792760010,v1:172.21.15.7:6801/3792760010] conn(0x7f9dc0072be0 0x7f9dc0075000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.135+0000 7f9dd0ff9640 20 client.4327.objecter _get_session s=0x7f9dc0072510 osd=1 3 2023-11-25T05:04:36.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.135+0000 7f9dd0ff9640 20 client.4327.objecter put_session s=0x7f9dc0072510 osd=1 3 2023-11-25T05:04:36.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.135+0000 7f9dd0ff9640 15 client.4327.objecter _session_command_op_remove -1 1 2023-11-25T05:04:36.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.135+0000 7f9dd0ff9640 20 client.4327.objecter _get_session s=0x7f9dc0072510 osd=1 3 2023-11-25T05:04:36.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.135+0000 7f9dd0ff9640 20 client.4327.objecter get_session s=0x7f9dc0072510 osd=1 3 2023-11-25T05:04:36.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.135+0000 7f9dd0ff9640 15 client.4327.objecter _session_command_op_assign 1 1 2023-11-25T05:04:36.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.135+0000 7f9dd0ff9640 20 client.4327.objecter put_session s=0x7f9dc0072510 osd=1 4 2023-11-25T05:04:36.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.135+0000 7f9dd0ff9640 10 client.4327.objecter _send_command 1 2023-11-25T05:04:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.135+0000 7f9dd0ff9640 1 -- 172.21.15.7:0/1503077748 --> [v2:172.21.15.7:6800/3792760010,v1:172.21.15.7:6801/3792760010] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9dc00756b0 con 0x7f9dc0072be0 2023-11-25T05:04:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.135+0000 7f9dd0ff9640 20 client.4327.objecter dump_active .. 0 homeless 2023-11-25T05:04:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.137+0000 7f9dd8c5f640 1 --2- 172.21.15.7:0/1503077748 >> [v2:172.21.15.7:6800/3792760010,v1:172.21.15.7:6801/3792760010] conn(0x7f9dc0072be0 0x7f9dc0075000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.137+0000 7f9dd8c5f640 10 monclient: get_auth_request con 0x7f9dc0072be0 auth_method 0 2023-11-25T05:04:36.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.142+0000 7f9dd8c5f640 1 --2- 172.21.15.7:0/1503077748 >> [v2:172.21.15.7:6800/3792760010,v1:172.21.15.7:6801/3792760010] conn(0x7f9dc0072be0 0x7f9dc0075000 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.142+0000 7f9dd0ff9640 10 client.4327.objecter ms_handle_connect 0x7f9dc0072be0 2023-11-25T05:04:36.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.145+0000 7f5f7b7fe640 1 -- 172.21.15.7:0/1898606146 <== osd.4 v2:172.21.15.90:6820/1576115380 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f5f800756d0 con 0x7f5f80072c00 2023-11-25T05:04:36.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.145+0000 7f5f7b7fe640 10 client.4296.objecter ms_dispatch 0x7f5f94000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:04:36.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.145+0000 7f5f7b7fe640 10 client.4296.objecter _finish_command 1 = system:0 2023-11-25T05:04:36.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.145+0000 7f5f7b7fe640 20 client.4296.objecter put_session s=0x7f5f80072530 osd=4 4 2023-11-25T05:04:36.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.145+0000 7f5f7b7fe640 15 client.4296.objecter _session_command_op_remove 4 1 2023-11-25T05:04:36.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.157+0000 7f465694e640 10 client.4321.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-25T05:04:36.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.157+0000 7f465694e640 15 client.4321.objecter _session_command_op_assign -1 2 2023-11-25T05:04:36.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.157+0000 7f465694e640 20 client.4321.objecter _get_session s=0x7f463c072530 osd=5 3 2023-11-25T05:04:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.157+0000 7f465694e640 20 client.4321.objecter put_session s=0x7f463c072530 osd=5 3 2023-11-25T05:04:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.157+0000 7f465694e640 20 client.4321.objecter _get_session s=0x7f463c072530 osd=5 3 2023-11-25T05:04:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.157+0000 7f465694e640 15 client.4321.objecter _session_command_op_remove -1 2 2023-11-25T05:04:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.157+0000 7f465694e640 20 client.4321.objecter get_session s=0x7f463c072530 osd=5 3 2023-11-25T05:04:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.157+0000 7f465694e640 15 client.4321.objecter _session_command_op_assign 5 2 2023-11-25T05:04:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.157+0000 7f465694e640 20 client.4321.objecter put_session s=0x7f463c072530 osd=5 4 2023-11-25T05:04:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.157+0000 7f465694e640 10 client.4321.objecter _send_command 2 2023-11-25T05:04:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.157+0000 7f465694e640 1 -- 172.21.15.7:0/1972786767 --> [v2:172.21.15.90:6810/132027349,v1:172.21.15.90:6811/132027349] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f46300011f0 con 0x7f463c072c00 2023-11-25T05:04:36.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f464cff9640 1 -- 172.21.15.7:0/1972786767 <== osd.5 v2:172.21.15.90:6810/132027349 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f46300011f0 con 0x7f463c072c00 2023-11-25T05:04:36.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f464cff9640 10 client.4321.objecter ms_dispatch 0x7f4650000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:04:36.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f464cff9640 10 client.4321.objecter _finish_command 2 = system:0 2023-11-25T05:04:36.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f464cff9640 20 client.4321.objecter put_session s=0x7f463c072530 osd=5 4 2023-11-25T05:04:36.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f464cff9640 15 client.4321.objecter _session_command_op_remove 5 2 2023-11-25T05:04:36.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f465694e640 10 librados: watch_flush enter 2023-11-25T05:04:36.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f465694e640 10 librados: watch_flush exit 2023-11-25T05:04:36.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f465694e640 10 client.4321.objecter close_session for osd.5 2023-11-25T05:04:36.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f465694e640 1 -- 172.21.15.7:0/1972786767 >> [v2:172.21.15.90:6810/132027349,v1:172.21.15.90:6811/132027349] conn(0x7f463c072c00 msgr2=0x7f463c075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f465694e640 1 --2- 172.21.15.7:0/1972786767 >> [v2:172.21.15.90:6810/132027349,v1:172.21.15.90:6811/132027349] conn(0x7f463c072c00 0x7f463c075020 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f465694e640 20 client.4321.objecter put_session s=0x7f463c072530 osd=5 1 2023-11-25T05:04:36.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f465694e640 20 client.4321.objecter shutdown clearing up homeless session... 2023-11-25T05:04:36.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f465694e640 10 client.4321.objecter successfully canceled tick 2023-11-25T05:04:36.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f5f995b0640 10 client.4296.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-25T05:04:36.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f5f995b0640 15 client.4296.objecter _session_command_op_assign -1 2 2023-11-25T05:04:36.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f5f995b0640 20 client.4296.objecter _get_session s=0x7f5f80072530 osd=4 3 2023-11-25T05:04:36.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f5f995b0640 20 client.4296.objecter put_session s=0x7f5f80072530 osd=4 3 2023-11-25T05:04:36.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f5f995b0640 20 client.4296.objecter _get_session s=0x7f5f80072530 osd=4 3 2023-11-25T05:04:36.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f5f995b0640 15 client.4296.objecter _session_command_op_remove -1 2 2023-11-25T05:04:36.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f5f995b0640 20 client.4296.objecter get_session s=0x7f5f80072530 osd=4 3 2023-11-25T05:04:36.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f5f995b0640 15 client.4296.objecter _session_command_op_assign 4 2 2023-11-25T05:04:36.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f5f995b0640 20 client.4296.objecter put_session s=0x7f5f80072530 osd=4 4 2023-11-25T05:04:36.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f5f995b0640 10 client.4296.objecter _send_command 2 2023-11-25T05:04:36.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.177+0000 7f5f995b0640 1 -- 172.21.15.7:0/1898606146 --> [v2:172.21.15.90:6820/1576115380,v1:172.21.15.90:6823/1576115380] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5f9413f3b0 con 0x7f5f80072c00 2023-11-25T05:04:36.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.178+0000 7f465694e640 1 -- 172.21.15.7:0/1972786767 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f463c06c820 msgr2=0x7f463c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.178+0000 7f465694e640 1 --2- 172.21.15.7:0/1972786767 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f463c06c820 0x7f463c06ece0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4640000f60 tx=0x7f46400079b0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.178+0000 7f465694e640 10 monclient: shutdown 2023-11-25T05:04:36.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.178+0000 7f465694e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.178+0000 7f465694e640 1 -- 172.21.15.7:0/1972786767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f465014a360 msgr2=0x7f465013bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.178+0000 7f465694e640 1 --2- 172.21.15.7:0/1972786767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f465014a360 0x7f465013bfa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f46440097a0 tx=0x7f46440046c0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.178+0000 7f4654eeb640 1 -- 172.21.15.7:0/1972786767 reap_dead start 2023-11-25T05:04:36.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.178+0000 7f465694e640 1 -- 172.21.15.7:0/1972786767 shutdown_connections 2023-11-25T05:04:36.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.178+0000 7f465694e640 1 -- 172.21.15.7:0/1972786767 >> 172.21.15.7:0/1972786767 conn(0x7f46500b01e0 msgr2=0x7f465006e920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.178+0000 7f465694e640 1 -- 172.21.15.7:0/1972786767 shutdown_connections 2023-11-25T05:04:36.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.178+0000 7f465694e640 1 -- 172.21.15.7:0/1972786767 wait complete. 2023-11-25T05:04:36.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.178+0000 7f465694e640 1 librados: shutdown 2023-11-25T05:04:36.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.184+0000 7f5f7b7fe640 1 -- 172.21.15.7:0/1898606146 <== osd.4 v2:172.21.15.90:6820/1576115380 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5f9413f3b0 con 0x7f5f80072c00 2023-11-25T05:04:36.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.184+0000 7f5f7b7fe640 10 client.4296.objecter ms_dispatch 0x7f5f94000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:04:36.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.184+0000 7f5f7b7fe640 10 client.4296.objecter _finish_command 2 = system:0 2023-11-25T05:04:36.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.184+0000 7f5f7b7fe640 20 client.4296.objecter put_session s=0x7f5f80072530 osd=4 4 2023-11-25T05:04:36.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.184+0000 7f5f7b7fe640 15 client.4296.objecter _session_command_op_remove 4 2 2023-11-25T05:04:36.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.185+0000 7f5f995b0640 10 librados: watch_flush enter 2023-11-25T05:04:36.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.188+0000 7f5f995b0640 10 librados: watch_flush exit 2023-11-25T05:04:36.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.188+0000 7f5f995b0640 10 client.4296.objecter close_session for osd.4 2023-11-25T05:04:36.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.188+0000 7f5f995b0640 1 -- 172.21.15.7:0/1898606146 >> [v2:172.21.15.90:6820/1576115380,v1:172.21.15.90:6823/1576115380] conn(0x7f5f80072c00 msgr2=0x7f5f80075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.188+0000 7f5f995b0640 1 --2- 172.21.15.7:0/1898606146 >> [v2:172.21.15.90:6820/1576115380,v1:172.21.15.90:6823/1576115380] conn(0x7f5f80072c00 0x7f5f80075020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.188+0000 7f5f995b0640 20 client.4296.objecter put_session s=0x7f5f80072530 osd=4 1 2023-11-25T05:04:36.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.188+0000 7f5f995b0640 20 client.4296.objecter shutdown clearing up homeless session... 2023-11-25T05:04:36.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.188+0000 7f5f995b0640 10 client.4296.objecter successfully canceled tick 2023-11-25T05:04:36.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.188+0000 7f5f995b0640 1 -- 172.21.15.7:0/1898606146 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f5f8006c820 msgr2=0x7f5f8006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.188+0000 7f5f995b0640 1 --2- 172.21.15.7:0/1898606146 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f5f8006c820 0x7f5f8006ece0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5f88000f60 tx=0x7f5f880079b0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.188+0000 7f5f995b0640 10 monclient: shutdown 2023-11-25T05:04:36.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.188+0000 7f5f995b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.188+0000 7f5f995b0640 1 -- 172.21.15.7:0/1898606146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f940a8e10 msgr2=0x7f5f94145da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.188+0000 7f5f995b0640 1 --2- 172.21.15.7:0/1898606146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f940a8e10 0x7f5f94145da0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5f940b8940 tx=0x7f5f84004730 comp rx=0 tx=0).stop 2023-11-25T05:04:36.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.188+0000 7f5f937fe640 1 -- 172.21.15.7:0/1898606146 reap_dead start 2023-11-25T05:04:36.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.190+0000 7f5f995b0640 1 -- 172.21.15.7:0/1898606146 shutdown_connections 2023-11-25T05:04:36.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.190+0000 7f5f995b0640 1 -- 172.21.15.7:0/1898606146 >> 172.21.15.7:0/1898606146 conn(0x7f5f94061a30 msgr2=0x7f5f9413cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.190+0000 7f5f995b0640 1 -- 172.21.15.7:0/1898606146 shutdown_connections 2023-11-25T05:04:36.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.190+0000 7f5f995b0640 1 -- 172.21.15.7:0/1898606146 wait complete. 2023-11-25T05:04:36.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.190+0000 7f5f995b0640 1 librados: shutdown 2023-11-25T05:04:36.196 INFO:teuthology.orchestra.run.smithi007.stdout:55834574852 2023-11-25T05:04:36.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-25T05:04:36.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.201+0000 7fa219ffb640 1 -- 172.21.15.7:0/1476158480 <== osd.2 v2:172.21.15.7:6808/1372486177 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fa2080756b0 con 0x7fa208072be0 2023-11-25T05:04:36.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.201+0000 7fa219ffb640 10 client.4324.objecter ms_dispatch 0x7fa224000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:04:36.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.201+0000 7fa219ffb640 10 client.4324.objecter _finish_command 1 = system:0 2023-11-25T05:04:36.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.201+0000 7fa219ffb640 20 client.4324.objecter put_session s=0x7fa208072510 osd=2 4 2023-11-25T05:04:36.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.201+0000 7fa219ffb640 15 client.4324.objecter _session_command_op_remove 2 1 2023-11-25T05:04:36.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.207+0000 7f9dd0ff9640 1 -- 172.21.15.7:0/1503077748 <== osd.1 v2:172.21.15.7:6800/3792760010 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9dc00756b0 con 0x7f9dc0072be0 2023-11-25T05:04:36.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.207+0000 7f9dd0ff9640 10 client.4327.objecter ms_dispatch 0x7f9dd4000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:04:36.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.207+0000 7f9dd0ff9640 10 client.4327.objecter _finish_command 1 = system:0 2023-11-25T05:04:36.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.207+0000 7f9dd0ff9640 20 client.4327.objecter put_session s=0x7f9dc0072510 osd=1 4 2023-11-25T05:04:36.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.207+0000 7f9dd0ff9640 15 client.4327.objecter _session_command_op_remove 1 1 2023-11-25T05:04:36.209 INFO:teuthology.orchestra.run.smithi007.stdout:64424509447 2023-11-25T05:04:36.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-25T05:04:36.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.237+0000 7fa22a36a640 10 client.4324.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-25T05:04:36.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.237+0000 7fa22a36a640 15 client.4324.objecter _session_command_op_assign -1 2 2023-11-25T05:04:36.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.237+0000 7fa22a36a640 20 client.4324.objecter _get_session s=0x7fa208072510 osd=2 3 2023-11-25T05:04:36.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.237+0000 7fa22a36a640 20 client.4324.objecter put_session s=0x7fa208072510 osd=2 3 2023-11-25T05:04:36.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.237+0000 7fa22a36a640 20 client.4324.objecter _get_session s=0x7fa208072510 osd=2 3 2023-11-25T05:04:36.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.237+0000 7fa22a36a640 15 client.4324.objecter _session_command_op_remove -1 2 2023-11-25T05:04:36.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.237+0000 7fa22a36a640 20 client.4324.objecter get_session s=0x7fa208072510 osd=2 3 2023-11-25T05:04:36.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.237+0000 7fa22a36a640 15 client.4324.objecter _session_command_op_assign 2 2 2023-11-25T05:04:36.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.237+0000 7fa22a36a640 20 client.4324.objecter put_session s=0x7fa208072510 osd=2 4 2023-11-25T05:04:36.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.237+0000 7fa22a36a640 10 client.4324.objecter _send_command 2 2023-11-25T05:04:36.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.237+0000 7fa22a36a640 1 -- 172.21.15.7:0/1476158480 --> [v2:172.21.15.7:6808/1372486177,v1:172.21.15.7:6809/1372486177] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa224075af0 con 0x7fa208072be0 2023-11-25T05:04:36.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.243+0000 7f9dda6c2640 10 client.4327.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-25T05:04:36.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.244+0000 7f9dda6c2640 15 client.4327.objecter _session_command_op_assign -1 2 2023-11-25T05:04:36.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.244+0000 7f9dda6c2640 20 client.4327.objecter _get_session s=0x7f9dc0072510 osd=1 3 2023-11-25T05:04:36.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.244+0000 7f9dda6c2640 20 client.4327.objecter put_session s=0x7f9dc0072510 osd=1 3 2023-11-25T05:04:36.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.244+0000 7f9dda6c2640 20 client.4327.objecter _get_session s=0x7f9dc0072510 osd=1 3 2023-11-25T05:04:36.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.244+0000 7f9dda6c2640 15 client.4327.objecter _session_command_op_remove -1 2 2023-11-25T05:04:36.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.244+0000 7f9dda6c2640 20 client.4327.objecter get_session s=0x7f9dc0072510 osd=1 3 2023-11-25T05:04:36.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.244+0000 7f9dda6c2640 15 client.4327.objecter _session_command_op_assign 1 2 2023-11-25T05:04:36.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.244+0000 7f9dda6c2640 20 client.4327.objecter put_session s=0x7f9dc0072510 osd=1 4 2023-11-25T05:04:36.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.244+0000 7f9dda6c2640 10 client.4327.objecter _send_command 2 2023-11-25T05:04:36.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.244+0000 7f9dda6c2640 1 -- 172.21.15.7:0/1503077748 --> [v2:172.21.15.7:6800/3792760010,v1:172.21.15.7:6801/3792760010] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9dd406fb20 con 0x7f9dc0072be0 2023-11-25T05:04:36.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.246+0000 7fa219ffb640 1 -- 172.21.15.7:0/1476158480 <== osd.2 v2:172.21.15.7:6808/1372486177 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa224075af0 con 0x7fa208072be0 2023-11-25T05:04:36.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.246+0000 7fa219ffb640 10 client.4324.objecter ms_dispatch 0x7fa224000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:04:36.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.246+0000 7fa219ffb640 10 client.4324.objecter _finish_command 2 = system:0 2023-11-25T05:04:36.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.246+0000 7fa219ffb640 20 client.4324.objecter put_session s=0x7fa208072510 osd=2 4 2023-11-25T05:04:36.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.246+0000 7fa219ffb640 15 client.4324.objecter _session_command_op_remove 2 2 2023-11-25T05:04:36.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 10 librados: watch_flush enter 2023-11-25T05:04:36.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 10 librados: watch_flush exit 2023-11-25T05:04:36.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 10 client.4324.objecter close_session for osd.2 2023-11-25T05:04:36.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 1 -- 172.21.15.7:0/1476158480 >> [v2:172.21.15.7:6808/1372486177,v1:172.21.15.7:6809/1372486177] conn(0x7fa208072be0 msgr2=0x7fa208075000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 1 --2- 172.21.15.7:0/1476158480 >> [v2:172.21.15.7:6808/1372486177,v1:172.21.15.7:6809/1372486177] conn(0x7fa208072be0 0x7fa208075000 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 20 client.4324.objecter put_session s=0x7fa208072510 osd=2 1 2023-11-25T05:04:36.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 20 client.4324.objecter shutdown clearing up homeless session... 2023-11-25T05:04:36.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 10 client.4324.objecter successfully canceled tick 2023-11-25T05:04:36.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 1 -- 172.21.15.7:0/1476158480 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fa20806c800 msgr2=0x7fa20806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 1 --2- 172.21.15.7:0/1476158480 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fa20806c800 0x7fa20806ecc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa2241545c0 tx=0x7fa214007a40 comp rx=0 tx=0).stop 2023-11-25T05:04:36.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 10 monclient: shutdown 2023-11-25T05:04:36.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 20 monclient: shutdown canceling and discarding version request 1 2023-11-25T05:04:36.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 1 -- 172.21.15.7:0/1476158480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2240a8830 msgr2=0x7fa224153c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 1 --2- 172.21.15.7:0/1476158480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2240a8830 0x7fa224153c30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa220033000 tx=0x7fa2200046c0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa229b69640 1 -- 172.21.15.7:0/1476158480 reap_dead start 2023-11-25T05:04:36.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 1 -- 172.21.15.7:0/1476158480 shutdown_connections 2023-11-25T05:04:36.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.248+0000 7fa22a36a640 1 -- 172.21.15.7:0/1476158480 >> 172.21.15.7:0/1476158480 conn(0x7fa2240618d0 msgr2=0x7fa224154e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.249+0000 7fa22a36a640 1 -- 172.21.15.7:0/1476158480 shutdown_connections 2023-11-25T05:04:36.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.249+0000 7fa22a36a640 1 -- 172.21.15.7:0/1476158480 wait complete. 2023-11-25T05:04:36.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.249+0000 7fa22a36a640 1 librados: shutdown 2023-11-25T05:04:36.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.253+0000 7f9dd0ff9640 1 -- 172.21.15.7:0/1503077748 <== osd.1 v2:172.21.15.7:6800/3792760010 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9dc8031b00 con 0x7f9dc0072be0 2023-11-25T05:04:36.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.253+0000 7f9dd0ff9640 10 client.4327.objecter ms_dispatch 0x7f9dd4000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:04:36.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.253+0000 7f9dd0ff9640 10 client.4327.objecter _finish_command 2 = system:0 2023-11-25T05:04:36.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.253+0000 7f9dd0ff9640 20 client.4327.objecter put_session s=0x7f9dc0072510 osd=1 4 2023-11-25T05:04:36.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.253+0000 7f9dd0ff9640 15 client.4327.objecter _session_command_op_remove 1 2 2023-11-25T05:04:36.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dba7fc640 10 librados: watch_flush enter 2023-11-25T05:04:36.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dba7fc640 10 librados: watch_flush exit 2023-11-25T05:04:36.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dba7fc640 10 client.4327.objecter close_session for osd.1 2023-11-25T05:04:36.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dba7fc640 1 -- 172.21.15.7:0/1503077748 >> [v2:172.21.15.7:6800/3792760010,v1:172.21.15.7:6801/3792760010] conn(0x7f9dc0072be0 msgr2=0x7f9dc0075000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dba7fc640 1 --2- 172.21.15.7:0/1503077748 >> [v2:172.21.15.7:6800/3792760010,v1:172.21.15.7:6801/3792760010] conn(0x7f9dc0072be0 0x7f9dc0075000 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dba7fc640 20 client.4327.objecter put_session s=0x7f9dc0072510 osd=1 1 2023-11-25T05:04:36.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dba7fc640 20 client.4327.objecter shutdown clearing up homeless session... 2023-11-25T05:04:36.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dba7fc640 10 client.4327.objecter successfully canceled tick 2023-11-25T05:04:36.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dba7fc640 1 -- 172.21.15.7:0/1503077748 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f9dc006c800 msgr2=0x7f9dc006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dba7fc640 1 --2- 172.21.15.7:0/1503077748 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f9dc006c800 0x7f9dc006ecc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc002070 tx=0x7f9dcc001660 comp rx=0 tx=0).stop 2023-11-25T05:04:36.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dba7fc640 10 monclient: shutdown 2023-11-25T05:04:36.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dba7fc640 20 monclient: shutdown canceling and discarding version request 1 2023-11-25T05:04:36.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dba7fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dba7fc640 1 -- 172.21.15.7:0/1503077748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd4143920 msgr2=0x7f9dd40b2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dba7fc640 1 --2- 172.21.15.7:0/1503077748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd4143920 0x7f9dd40b2e20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9dc400e850 tx=0x7f9dc400ed10 comp rx=0 tx=0).stop 2023-11-25T05:04:36.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.255+0000 7f9dd8c5f640 1 -- 172.21.15.7:0/1503077748 reap_dead start 2023-11-25T05:04:36.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.256+0000 7f9dba7fc640 1 -- 172.21.15.7:0/1503077748 shutdown_connections 2023-11-25T05:04:36.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.256+0000 7f9dba7fc640 1 -- 172.21.15.7:0/1503077748 >> 172.21.15.7:0/1503077748 conn(0x7f9dd40b01e0 msgr2=0x7f9dd406e920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.257+0000 7f9dba7fc640 1 -- 172.21.15.7:0/1503077748 shutdown_connections 2023-11-25T05:04:36.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.258+0000 7f9dba7fc640 1 -- 172.21.15.7:0/1503077748 wait complete. 2023-11-25T05:04:36.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.258+0000 7f9dba7fc640 1 librados: shutdown 2023-11-25T05:04:36.269 INFO:teuthology.orchestra.run.smithi007.stdout:68719476741 2023-11-25T05:04:36.270 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-25T05:04:36.279 INFO:teuthology.orchestra.run.smithi007.stdout:68719476742 2023-11-25T05:04:36.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-25T05:04:36.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.286+0000 7f26a1d01640 10 monclient: get_monmap_and_config 2023-11-25T05:04:36.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.286+0000 7f26a1d01640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.287+0000 7f26a1d01640 10 monclient: monmap: 2023-11-25T05:04:36.289 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.289 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.289 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:36.287936+0000 2023-11-25T05:04:36.289 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:36.287936+0000 2023-11-25T05:04:36.289 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.289 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.289 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:36.289 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:36.289 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:36.289 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:36.289 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:36.290 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.288+0000 7ff74487c640 10 monclient: get_monmap_and_config 2023-11-25T05:04:36.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.288+0000 7ff74487c640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.288+0000 7ff74487c640 10 monclient: monmap: 2023-11-25T05:04:36.292 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.292 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.292 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:36.289045+0000 2023-11-25T05:04:36.292 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:36.289045+0000 2023-11-25T05:04:36.292 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.292 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.292 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:36.292 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:36.292 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:36.292 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:36.292 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:36.293 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.287+0000 7f26a1d01640 1 Processor -- start 2023-11-25T05:04:36.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.288+0000 7f26a1d01640 1 -- start start 2023-11-25T05:04:36.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.288+0000 7f26a1d01640 10 monclient: init 2023-11-25T05:04:36.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.288+0000 7f26a1d01640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.288+0000 7f26a1d01640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-11-25T05:04:36.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.289+0000 7f26a1d01640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f269c0a8a30 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:36.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.289+0000 7f26a1d01640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f269c142bf0 0x7f269c13ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.289+0000 7f26a1d01640 10 monclient(hunting): picked mon.noname-c con 0x7f269c142bf0 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:36.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.289+0000 7f26a1d01640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f269c0a8e10 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:36.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.289+0000 7f26a1d01640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269c0aac10 con 0x7f269c0a8e10 2023-11-25T05:04:36.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.289+0000 7f26a1d01640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269c0aaef0 con 0x7f269c0a8a30 2023-11-25T05:04:36.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.289+0000 7f26a1d01640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.289+0000 7f26a1d01640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f269c0aaa10 con 0x7f269c142bf0 2023-11-25T05:04:36.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.289+0000 7f26a1d01640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.289+0000 7f26a1d01640 10 monclient(hunting): authenticate will time out at 1402.531936s 2023-11-25T05:04:36.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.289+0000 7ff74487c640 1 Processor -- start 2023-11-25T05:04:36.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.289+0000 7f269affd640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f269c142bf0 0x7f269c13ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7f269bfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f269c0a8e10 0x7f269c13f230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37580/0 (socket says 172.21.15.7:37580) 2023-11-25T05:04:36.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7f269bfff640 1 -- 172.21.15.7:0/2213828578 learned_addr learned my addr 172.21.15.7:0/2213828578 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:36.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.289+0000 7ff74487c640 1 -- start start 2023-11-25T05:04:36.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.289+0000 7ff74487c640 10 monclient: init 2023-11-25T05:04:36.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7ff74487c640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7ff74487c640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-11-25T05:04:36.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7ff74487c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff7400b0f20 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:36.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7ff74487c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff7400b29e0 0x7ff740143240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7ff74487c640 10 monclient(hunting): picked mon.noname-a con 0x7ff7400b29e0 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:36.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7ff74487c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff740143780 0x7ff740150c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7ff74487c640 10 monclient(hunting): picked mon.noname-b con 0x7ff740143780 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7ff74487c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff74006dc60 con 0x7ff7400b0f20 2023-11-25T05:04:36.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7ff74487c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7ff74487c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff740146cf0 con 0x7ff7400b29e0 2023-11-25T05:04:36.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7ff74487c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7ff74487c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7400b1300 con 0x7ff740143780 2023-11-25T05:04:36.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7ff74487c640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7ff74487c640 10 monclient(hunting): authenticate will time out at 1402.533677s 2023-11-25T05:04:36.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7f269affd640 10 monclient(hunting): get_auth_request con 0x7f269c142bf0 auth_method 0 2023-11-25T05:04:36.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7f269affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7f269affd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.290+0000 7f269affd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.291+0000 7f269affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.291+0000 7f269affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.291+0000 7f269affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.293+0000 7ff73ed9d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff740143780 0x7ff740150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.293+0000 7ff73ed9d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff740143780 0x7ff740150c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33688/0 (socket says 172.21.15.7:33688) 2023-11-25T05:04:36.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.293+0000 7ff73ed9d640 1 -- 172.21.15.7:0/432146691 learned_addr learned my addr 172.21.15.7:0/432146691 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.293+0000 7ff73ed9d640 10 monclient(hunting): get_auth_request con 0x7ff740143780 auth_method 0 2023-11-25T05:04:36.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.293+0000 7ff73ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.293+0000 7ff73ed9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.293+0000 7ff73ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.295+0000 7ff73ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.295+0000 7ff73ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.295+0000 7ff73ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.293+0000 7f269affd640 10 monclient(hunting): handle_auth_done global_id 4330 payload 274 2023-11-25T05:04:36.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.293+0000 7f269affd640 1 -- 172.21.15.7:0/2213828578 >> v1:172.21.15.90:6789/0 conn(0x7f269c0a8a30 legacy=0x7f269c1424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.293+0000 7f269affd640 1 -- 172.21.15.7:0/2213828578 >> v1:172.21.15.7:6789/0 conn(0x7f269c0a8e10 legacy=0x7f269c13f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.293+0000 7f269affd640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.293+0000 7f269affd640 1 monclient: found mon.noname-c 2023-11-25T05:04:36.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.293+0000 7f269affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.293+0000 7f269affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.293+0000 7f269affd640 1 -- 172.21.15.7:0/2213828578 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269c0a97f0 con 0x7f269c142bf0 2023-11-25T05:04:36.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.298+0000 7f269affd640 1 --2- 172.21.15.7:0/2213828578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f269c142bf0 0x7f269c13ecf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f268c0097a0 tx=0x7f268c02ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a08ff722152d18d server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.303+0000 7f269a7fc640 1 -- 172.21.15.7:0/2213828578 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c02f840 con 0x7f269c142bf0 2023-11-25T05:04:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.303+0000 7f269a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.303+0000 7f269a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.303+0000 7f269a7fc640 10 monclient: dump: 2023-11-25T05:04:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.306 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.303+0000 7f269a7fc640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.303+0000 7f269a7fc640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.303+0000 7f269a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.303+0000 7f26a1d01640 5 monclient: authenticate success, global_id 4330 2023-11-25T05:04:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.303+0000 7f26a1d01640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:36.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.302+0000 7ff73ed9d640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2023-11-25T05:04:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.302+0000 7ff73ed9d640 1 -- 172.21.15.7:0/432146691 >> v2:172.21.15.7:3300/0 conn(0x7ff7400b29e0 msgr2=0x7ff740143240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.302+0000 7ff73ed9d640 1 --2- 172.21.15.7:0/432146691 >> v2:172.21.15.7:3300/0 conn(0x7ff7400b29e0 0x7ff740143240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.302+0000 7ff73ed9d640 1 -- 172.21.15.7:0/432146691 >> v1:172.21.15.7:6789/0 conn(0x7ff7400b0f20 legacy=0x7ff7400b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.302+0000 7ff73ed9d640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.302+0000 7ff73ed9d640 1 monclient: found mon.noname-b 2023-11-25T05:04:36.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.302+0000 7ff73ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.302+0000 7ff73ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.302+0000 7ff73ed9d640 1 -- 172.21.15.7:0/432146691 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74006e800 con 0x7ff740143780 2023-11-25T05:04:36.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.303+0000 7f269a7fc640 1 -- 172.21.15.7:0/2213828578 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268c02f9a0 con 0x7f269c142bf0 2023-11-25T05:04:36.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.303+0000 7f269a7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.303+0000 7f26a1d01640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:36.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.303+0000 7f26a1d01640 4 set_mon_vals no callback set 2023-11-25T05:04:36.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.304+0000 7f26a1d01640 10 monclient: shutdown 2023-11-25T05:04:36.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.304+0000 7f26a1d01640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.304+0000 7f26a1d01640 1 -- 172.21.15.7:0/2213828578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f269c142bf0 msgr2=0x7f269c13ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.304+0000 7f269a7fc640 1 -- 172.21.15.7:0/2213828578 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c02f840 con 0x7f269c142bf0 2023-11-25T05:04:36.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.304+0000 7f26a1d01640 1 --2- 172.21.15.7:0/2213828578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f269c142bf0 0x7f269c13ecf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f268c0097a0 tx=0x7f268c02ba20 comp rx=0 tx=0).stop 2023-11-25T05:04:36.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.304+0000 7f269a7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2023-11-25T05:04:36.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.306+0000 7f26a1d01640 1 -- 172.21.15.7:0/2213828578 shutdown_connections 2023-11-25T05:04:36.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.306+0000 7f26a1d01640 1 --2- 172.21.15.7:0/2213828578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f269c142bf0 0x7f269c13ecf0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.306+0000 7f26a1d01640 1 -- 172.21.15.7:0/2213828578 >> 172.21.15.7:0/2213828578 conn(0x7f269c061a30 msgr2=0x7f269c0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.306+0000 7f26a1d01640 1 -- 172.21.15.7:0/2213828578 shutdown_connections 2023-11-25T05:04:36.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.306+0000 7ff73ed9d640 1 --2- 172.21.15.7:0/432146691 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff740143780 0x7ff740150c80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff738009e50 tx=0x7ff73802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22de32c748372996 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.306+0000 7f26a1d01640 1 -- 172.21.15.7:0/2213828578 wait complete. 2023-11-25T05:04:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.307+0000 7f26a1d01640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.307+0000 7f26a1d01640 10 monclient: monmap: 2023-11-25T05:04:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.314 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:36.314 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:36.314 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:36.314 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.307+0000 7f26a1d01640 1 librados: starting msgr at 2023-11-25T05:04:36.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.307+0000 7f26a1d01640 1 librados: starting objecter 2023-11-25T05:04:36.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.307+0000 7f26a1d01640 1 Processor -- start 2023-11-25T05:04:36.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.308+0000 7f26a1d01640 1 -- start start 2023-11-25T05:04:36.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.308+0000 7f26a1d01640 1 librados: setting wanted keys 2023-11-25T05:04:36.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.308+0000 7f26a1d01640 1 librados: calling monclient init 2023-11-25T05:04:36.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.308+0000 7f26a1d01640 10 monclient: init 2023-11-25T05:04:36.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-25T05:04:36.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f269c0a8a30 0x7f269c0b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 10 monclient(hunting): picked mon.noname-a con 0x7f269c0a8a30 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:36.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f269c0a8e10 0x7f269c145980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 10 monclient(hunting): picked mon.noname-b con 0x7f269c0a8e10 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:36.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f269c142bf0 0x7f269c0b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 10 monclient(hunting): picked mon.noname-c con 0x7f269c142bf0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c0ac360 con 0x7f269c0a8a30 2023-11-25T05:04:36.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c145fa0 con 0x7f269c0a8e10 2023-11-25T05:04:36.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f269c145300 con 0x7f269c142bf0 2023-11-25T05:04:36.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.309+0000 7f26a1d01640 10 monclient(hunting): authenticate will time out at 1402.552134s 2023-11-25T05:04:36.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.311+0000 7f269bfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f269c142bf0 0x7f269c0b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.311+0000 7f269bfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f269c142bf0 0x7f269c0b0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33704/0 (socket says 172.21.15.7:33704) 2023-11-25T05:04:36.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.311+0000 7f269bfff640 1 -- 172.21.15.7:0/3597088794 learned_addr learned my addr 172.21.15.7:0/3597088794 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.311+0000 7f269bfff640 10 monclient(hunting): get_auth_request con 0x7f269c142bf0 auth_method 0 2023-11-25T05:04:36.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.311+0000 7f269bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.311+0000 7f269bfff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.311+0000 7f269bfff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.311+0000 7f269b7fe640 1 --2- 172.21.15.7:0/3597088794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f269c0a8a30 0x7f269c0b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.311+0000 7f269b7fe640 10 monclient(hunting): get_auth_request con 0x7f269c0a8a30 auth_method 0 2023-11-25T05:04:36.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.311+0000 7f269b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.311+0000 7f269b7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.311+0000 7f269b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.312+0000 7f269bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.312+0000 7f269bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.312+0000 7f269bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.312+0000 7f269b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.312+0000 7f269b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.312+0000 7f269b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.314+0000 7f269affd640 1 --2- 172.21.15.7:0/3597088794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f269c0a8e10 0x7f269c145980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.315+0000 7f269affd640 10 monclient(hunting): get_auth_request con 0x7f269c0a8e10 auth_method 0 2023-11-25T05:04:36.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.315+0000 7f269affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.315+0000 7f269affd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.315+0000 7f269affd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.315+0000 7f269affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.315+0000 7f269affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.315+0000 7f269affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.319+0000 7f269b7fe640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2023-11-25T05:04:36.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.319+0000 7f269b7fe640 1 -- 172.21.15.7:0/3597088794 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f269c142bf0 msgr2=0x7f269c0b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.319+0000 7f269b7fe640 1 --2- 172.21.15.7:0/3597088794 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f269c142bf0 0x7f269c0b0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.319+0000 7f269b7fe640 1 -- 172.21.15.7:0/3597088794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f269c0a8e10 msgr2=0x7f269c145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.319+0000 7f269b7fe640 1 --2- 172.21.15.7:0/3597088794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f269c0a8e10 0x7f269c145980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.319+0000 7f269b7fe640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.319+0000 7f269b7fe640 1 monclient: found mon.noname-a 2023-11-25T05:04:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.319+0000 7f269b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.319+0000 7f269b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.319+0000 7f269b7fe640 1 -- 172.21.15.7:0/3597088794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c009400 con 0x7f269c0a8a30 2023-11-25T05:04:36.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.324+0000 7f269b7fe640 1 --2- 172.21.15.7:0/3597088794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f269c0a8a30 0x7f269c0b6e40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f269000c390 tx=0x7f269000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.324+0000 7f2683fff640 10 client.?.objecter ms_handle_connect 0x7f269c0a8a30 2023-11-25T05:04:36.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.324+0000 7f2683fff640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:36.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f77b5f6c640 10 monclient: get_monmap_and_config 2023-11-25T05:04:36.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f77b5f6c640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f77b5f6c640 10 monclient: monmap: 2023-11-25T05:04:36.332 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.332 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.332 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:36.331088+0000 2023-11-25T05:04:36.332 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:36.331088+0000 2023-11-25T05:04:36.332 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.332 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.332 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:36.332 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:36.332 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:36.333 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f2683fff640 1 -- 172.21.15.7:0/3597088794 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26900052d0 con 0x7f269c0a8a30 2023-11-25T05:04:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f2683fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f2683fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f2683fff640 10 monclient: dump: 2023-11-25T05:04:36.334 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.334 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.334 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.334 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.334 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.334 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.334 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.334 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.334 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.335 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f2683fff640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f2683fff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f2683fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f2683fff640 1 -- 172.21.15.7:0/3597088794 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2690005430 con 0x7f269c0a8a30 2023-11-25T05:04:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f2683fff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f2683fff640 1 -- 172.21.15.7:0/3597088794 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690003830 con 0x7f269c0a8a30 2023-11-25T05:04:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f2683fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f2683fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f2683fff640 10 monclient: dump: 2023-11-25T05:04:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.337 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f26a1d01640 5 monclient: authenticate success, global_id 4305 2023-11-25T05:04:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f26a1d01640 10 monclient: _renew_subs 2023-11-25T05:04:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.331+0000 7f77b5f6c640 1 Processor -- start 2023-11-25T05:04:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f26a1d01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f26a1d01640 1 -- 172.21.15.7:0/3597088794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269c1454c0 con 0x7f269c0a8a30 2023-11-25T05:04:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.331+0000 7f77b5f6c640 1 -- start start 2023-11-25T05:04:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.331+0000 7f77b5f6c640 10 monclient: init 2023-11-25T05:04:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.332+0000 7f77b5f6c640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.332+0000 7f77b5f6c640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-11-25T05:04:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.330+0000 7f2699ffb640 4 set_mon_vals no callback set 2023-11-25T05:04:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.331+0000 7f26a1d01640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.332+0000 7f77b5f6c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f77a8038d80 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.333+0000 7f77b5f6c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f77a8039160 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.333+0000 7f77b5f6c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f77a80c8a30 0x7f77a80d3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.331+0000 7f26a1d01640 10 monclient: _renew_subs 2023-11-25T05:04:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.331+0000 7f26a1d01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.331+0000 7f26a1d01640 1 -- 172.21.15.7:0/3597088794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269c0b1310 con 0x7f269c0a8a30 2023-11-25T05:04:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.333+0000 7f77b5f6c640 10 monclient(hunting): picked mon.noname-a con 0x7f77a80c8a30 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.333+0000 7f77b5f6c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a80c7e70 con 0x7f77a8039160 2023-11-25T05:04:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.333+0000 7f77b5f6c640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a8035440 con 0x7f77a8038d80 2023-11-25T05:04:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.333+0000 7f77b5f6c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.333+0000 7f77b5f6c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a8037610 con 0x7f77a80c8a30 2023-11-25T05:04:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.333+0000 7f77b5f6c640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.333+0000 7f77b5f6c640 10 monclient(hunting): authenticate will time out at 1402.576207s 2023-11-25T05:04:36.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.331+0000 7f26a1d01640 1 librados: init done 2023-11-25T05:04:36.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.333+0000 7f26a1d01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.333+0000 7f26a1d01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.333+0000 7f26a1d01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.333+0000 7f26a1d01640 1 -- 172.21.15.7:0/3597088794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269c067ac0 con 0x7f269c0a8a30 2023-11-25T05:04:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.333+0000 7f77b4f6a640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f77a8038d80 0x7f77a80c5230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:52616/0 (socket says 172.21.15.7:52616) 2023-11-25T05:04:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.333+0000 7f77b4f6a640 1 -- 172.21.15.7:0/4120082866 learned_addr learned my addr 172.21.15.7:0/4120082866 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:36.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.346+0000 7f77b576b640 1 --2- 172.21.15.7:0/4120082866 >> v2:172.21.15.7:3300/0 conn(0x7f77a80c8a30 0x7f77a80d3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.346+0000 7f77b576b640 10 monclient(hunting): get_auth_request con 0x7f77a80c8a30 auth_method 0 2023-11-25T05:04:36.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.346+0000 7f77b576b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.346+0000 7f77b576b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.346+0000 7f77b576b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.360+0000 7fd3a6a0e640 10 monclient: get_monmap_and_config 2023-11-25T05:04:36.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.360+0000 7fd3a6a0e640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.360+0000 7fd3a6a0e640 10 monclient: monmap: 2023-11-25T05:04:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:36.361109+0000 2023-11-25T05:04:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:36.361109+0000 2023-11-25T05:04:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:36.363 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:36.363 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:36.363 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 1 Processor -- start 2023-11-25T05:04:36.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 1 -- start start 2023-11-25T05:04:36.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 10 monclient: init 2023-11-25T05:04:36.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2023-11-25T05:04:36.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3a00a8a30 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:36.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd3a0142bf0 0x7fd3a013ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 10 monclient(hunting): picked mon.noname-a con 0x7fd3a0142bf0 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:36.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd3a00a8e10 0x7fd3a013f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 10 monclient(hunting): picked mon.noname-c con 0x7fd3a00a8e10 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:36.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a00aac10 con 0x7fd3a00a8a30 2023-11-25T05:04:36.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a00aaa10 con 0x7fd3a0142bf0 2023-11-25T05:04:36.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a00aaef0 con 0x7fd3a00a8e10 2023-11-25T05:04:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.361+0000 7fd3a6a0e640 10 monclient(hunting): authenticate will time out at 1402.604782s 2023-11-25T05:04:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.362+0000 7fd39ffff640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd3a00a8a30 0x7fd3a01424e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:52622/0 (socket says 172.21.15.7:52622) 2023-11-25T05:04:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.362+0000 7fd39ffff640 1 -- 172.21.15.7:0/3417074385 learned_addr learned my addr 172.21.15.7:0/3417074385 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.362+0000 7fd3a4fab640 1 --2- 172.21.15.7:0/3417074385 >> v2:172.21.15.90:3300/0 conn(0x7fd3a00a8e10 0x7fd3a013f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.363+0000 7fd3a4fab640 10 monclient(hunting): get_auth_request con 0x7fd3a00a8e10 auth_method 0 2023-11-25T05:04:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.363+0000 7fd3a4fab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.363+0000 7fd3a4fab640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.363+0000 7fd3a4fab640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.364+0000 7fd3a4fab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.364+0000 7fd3a4fab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.364+0000 7fd3a4fab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.366+0000 7fd3a4fab640 10 monclient(hunting): handle_auth_done global_id 4336 payload 274 2023-11-25T05:04:36.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.366+0000 7fd3a4fab640 1 -- 172.21.15.7:0/3417074385 >> v2:172.21.15.7:3300/0 conn(0x7fd3a0142bf0 msgr2=0x7fd3a013ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:36.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.366+0000 7fd3a4fab640 1 --2- 172.21.15.7:0/3417074385 >> v2:172.21.15.7:3300/0 conn(0x7fd3a0142bf0 0x7fd3a013ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.366+0000 7fd3a4fab640 1 -- 172.21.15.7:0/3417074385 >> v1:172.21.15.90:6789/0 conn(0x7fd3a00a8a30 legacy=0x7fd3a01424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.366+0000 7fd3a4fab640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.366+0000 7fd3a4fab640 1 monclient: found mon.noname-c 2023-11-25T05:04:36.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.366+0000 7fd3a4fab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.366+0000 7fd3a4fab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.366+0000 7fd3a4fab640 1 -- 172.21.15.7:0/3417074385 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a00a97f0 con 0x7fd3a00a8e10 2023-11-25T05:04:36.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.369+0000 7fd3a4fab640 1 --2- 172.21.15.7:0/3417074385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a00a8e10 0x7fd3a013f230 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd3a00b3a60 tx=0x7fd39802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71f33732e58ecadf server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd39effd640 1 -- 172.21.15.7:0/3417074385 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39802cd00 con 0x7fd3a00a8e10 2023-11-25T05:04:36.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.372+0000 7f2683fff640 1 -- 172.21.15.7:0/3597088794 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2690003990 con 0x7f269c0a8a30 2023-11-25T05:04:36.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7f2683fff640 1 --2- 172.21.15.7:0/3597088794 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f268806c850 0x7f268806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd39effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd39effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:36.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd39effd640 10 monclient: dump: 2023-11-25T05:04:36.377 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.377 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.377 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.377 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.378 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd39effd640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd39effd640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd39effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd39effd640 1 -- 172.21.15.7:0/3417074385 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39802ce60 con 0x7fd3a00a8e10 2023-11-25T05:04:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd39effd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd39effd640 1 -- 172.21.15.7:0/3417074385 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398034660 con 0x7fd3a00a8e10 2023-11-25T05:04:36.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd39effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd39effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:36.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd39effd640 10 monclient: dump: 2023-11-25T05:04:36.379 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.379 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.379 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.379 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.379 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.379 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.379 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.379 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.379 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.380 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd3a6a0e640 5 monclient: authenticate success, global_id 4336 2023-11-25T05:04:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd3a6a0e640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.373+0000 7fd3a6a0e640 4 set_mon_vals no callback set 2023-11-25T05:04:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.374+0000 7f269affd640 1 --2- 172.21.15.7:0/3597088794 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f268806c850 0x7f268806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.374+0000 7f269affd640 10 monclient: get_auth_request con 0x7f268806c850 auth_method 0 2023-11-25T05:04:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.374+0000 7fd3a6a0e640 10 monclient: shutdown 2023-11-25T05:04:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.374+0000 7fd3a6a0e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.374+0000 7fd3a6a0e640 1 -- 172.21.15.7:0/3417074385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a00a8e10 msgr2=0x7fd3a013f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.374+0000 7fd3a6a0e640 1 --2- 172.21.15.7:0/3417074385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a00a8e10 0x7fd3a013f230 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd3a00b3a60 tx=0x7fd39802bf50 comp rx=0 tx=0).stop 2023-11-25T05:04:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.374+0000 7fd3a6a0e640 1 -- 172.21.15.7:0/3417074385 shutdown_connections 2023-11-25T05:04:36.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.374+0000 7fd3a6a0e640 1 --2- 172.21.15.7:0/3417074385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a00a8e10 0x7fd3a013f230 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.374+0000 7fd3a6a0e640 1 --2- 172.21.15.7:0/3417074385 >> v2:172.21.15.7:3300/0 conn(0x7fd3a0142bf0 0x7fd3a013ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.374+0000 7fd3a6a0e640 1 -- 172.21.15.7:0/3417074385 >> 172.21.15.7:0/3417074385 conn(0x7fd3a0061a30 msgr2=0x7fd3a00b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.375+0000 7f269affd640 1 --2- 172.21.15.7:0/3597088794 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f268806c850 0x7f268806ed10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f268c036000 tx=0x7f268c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.375+0000 7f2683fff640 10 client.4305.objecter ms_handle_connect 0x7f268806c850 2023-11-25T05:04:36.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.374+0000 7fd3a6a0e640 1 -- 172.21.15.7:0/3417074385 shutdown_connections 2023-11-25T05:04:36.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.374+0000 7fd3a6a0e640 1 -- 172.21.15.7:0/3417074385 wait complete. 2023-11-25T05:04:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.376+0000 7fd3a6a0e640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.376+0000 7fd3a6a0e640 10 monclient: monmap: 2023-11-25T05:04:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:36.384 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.376+0000 7fd3a6a0e640 1 librados: starting msgr at 2023-11-25T05:04:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.376+0000 7fd3a6a0e640 1 librados: starting objecter 2023-11-25T05:04:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.377+0000 7fd3a6a0e640 1 Processor -- start 2023-11-25T05:04:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.377+0000 7fd3a6a0e640 1 -- start start 2023-11-25T05:04:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.377+0000 7fd3a6a0e640 1 librados: setting wanted keys 2023-11-25T05:04:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.377+0000 7fd3a6a0e640 1 librados: calling monclient init 2023-11-25T05:04:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.377+0000 7fd3a6a0e640 10 monclient: init 2023-11-25T05:04:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.378+0000 7f77b576b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.378+0000 7f77b576b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.378+0000 7f77b576b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.378+0000 7fd3a6a0e640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.378+0000 7fd3a6a0e640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-25T05:04:36.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.378+0000 7fd3a6a0e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a00a8a30 0x7fd3a0145500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.378+0000 7fd3a6a0e640 10 monclient(hunting): picked mon.noname-b con 0x7fd3a00a8a30 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:36.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.378+0000 7fd3a6a0e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3a00a8e10 0x7fd3a00ad060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.379+0000 7fd3a6a0e640 10 monclient(hunting): picked mon.noname-c con 0x7fd3a00a8e10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.379+0000 7fd3a6a0e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a0142bf0 0x7fd3a00ad680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.379+0000 7fd3a6a0e640 10 monclient(hunting): picked mon.noname-a con 0x7fd3a0142bf0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:36.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.379+0000 7fd3a6a0e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.379+0000 7fd3a6a0e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a00add50 con 0x7fd3a0142bf0 2023-11-25T05:04:36.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.379+0000 7fd3a6a0e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.379+0000 7fd3a6a0e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a01d7000 con 0x7fd3a00a8a30 2023-11-25T05:04:36.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.379+0000 7fd3a6a0e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.379+0000 7fd3a6a0e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a01d71c0 con 0x7fd3a00a8e10 2023-11-25T05:04:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.379+0000 7fd3a6a0e640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.379+0000 7fd3a6a0e640 10 monclient(hunting): authenticate will time out at 1402.622288s 2023-11-25T05:04:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.380+0000 7fd39ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a00a8a30 0x7fd3a0145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.380+0000 7fd39ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a00a8a30 0x7fd3a0145500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:38430/0 (socket says 172.21.15.7:38430) 2023-11-25T05:04:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.380+0000 7fd39ffff640 1 -- 172.21.15.7:0/758239465 learned_addr learned my addr 172.21.15.7:0/758239465 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.380+0000 7fd39ffff640 10 monclient(hunting): get_auth_request con 0x7fd3a00a8a30 auth_method 0 2023-11-25T05:04:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.380+0000 7fd39ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.380+0000 7fd39ffff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.380+0000 7fd39ffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.380+0000 7f2683fff640 1 -- 172.21.15.7:0/3597088794 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2690057d40 con 0x7f269c0a8a30 2023-11-25T05:04:36.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.380+0000 7f2683fff640 10 client.4305.objecter ms_dispatch 0x7f269c000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:36.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.380+0000 7f2683fff640 3 client.4305.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:36.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.380+0000 7f2683fff640 3 client.4305.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:36.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.380+0000 7f2683fff640 20 client.4305.objecter dump_active .. 0 homeless 2023-11-25T05:04:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.380+0000 7fd39ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.380+0000 7fd39ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.381+0000 7fd39ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.381+0000 7fd3a4fab640 1 --2- 172.21.15.7:0/758239465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a0142bf0 0x7fd3a00ad680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7fd39ffff640 10 monclient(hunting): handle_auth_done global_id 4339 payload 824 2023-11-25T05:04:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7fd39ffff640 1 -- 172.21.15.7:0/758239465 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3a00a8e10 msgr2=0x7fd3a00ad060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7fd39ffff640 1 --2- 172.21.15.7:0/758239465 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3a00a8e10 0x7fd3a00ad060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7fd39ffff640 1 -- 172.21.15.7:0/758239465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a0142bf0 msgr2=0x7fd3a00ad680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7fd39ffff640 1 --2- 172.21.15.7:0/758239465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a0142bf0 0x7fd3a00ad680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7fd39ffff640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7fd39ffff640 1 monclient: found mon.noname-b 2023-11-25T05:04:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7fd39ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7fd39ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7fd39ffff640 1 -- 172.21.15.7:0/758239465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd398009a70 con 0x7fd3a00a8a30 2023-11-25T05:04:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7f77b576b640 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2023-11-25T05:04:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7f77b576b640 1 -- 172.21.15.7:0/4120082866 >> v1:172.21.15.90:6789/0 conn(0x7f77a8038d80 legacy=0x7f77a80c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7f77b576b640 1 -- 172.21.15.7:0/4120082866 >> v1:172.21.15.7:6789/0 conn(0x7f77a8039160 legacy=0x7f77a80c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7f77b576b640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7f77b576b640 1 monclient: found mon.noname-a 2023-11-25T05:04:36.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7f77b576b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7f77b576b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.383+0000 7f77b576b640 1 -- 172.21.15.7:0/4120082866 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a80c5940 con 0x7f77a80c8a30 2023-11-25T05:04:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.386+0000 7fd39ffff640 1 --2- 172.21.15.7:0/758239465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a00a8a30 0x7fd3a0145500 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd39000c4a0 tx=0x7fd39000c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.386+0000 7fd39cff9640 10 client.?.objecter ms_handle_connect 0x7fd3a00a8a30 2023-11-25T05:04:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.386+0000 7fd39cff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd39cff9640 1 -- 172.21.15.7:0/758239465 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390005ce0 con 0x7fd3a00a8a30 2023-11-25T05:04:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd39cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd39cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd39cff9640 10 monclient: dump: 2023-11-25T05:04:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.397 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd39cff9640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd39cff9640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd39cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd39cff9640 1 -- 172.21.15.7:0/758239465 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd390005e40 con 0x7fd3a00a8a30 2023-11-25T05:04:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd3a6a0e640 5 monclient: authenticate success, global_id 4339 2023-11-25T05:04:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd3a6a0e640 10 monclient: _renew_subs 2023-11-25T05:04:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd3a6a0e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd3a6a0e640 1 -- 172.21.15.7:0/758239465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a01d73e0 con 0x7fd3a00a8a30 2023-11-25T05:04:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd3a6a0e640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd39cff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd39cff9640 1 -- 172.21.15.7:0/758239465 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390007270 con 0x7fd3a00a8a30 2023-11-25T05:04:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd39cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd39e7fc640 4 set_mon_vals no callback set 2023-11-25T05:04:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd39cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.390+0000 7fd39cff9640 10 monclient: dump: 2023-11-25T05:04:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.406 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.391+0000 7fd3a6a0e640 10 monclient: _renew_subs 2023-11-25T05:04:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.391+0000 7fd3a6a0e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.391+0000 7fd3a6a0e640 1 -- 172.21.15.7:0/758239465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a01d7890 con 0x7fd3a00a8a30 2023-11-25T05:04:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.391+0000 7fd3a6a0e640 1 librados: init done 2023-11-25T05:04:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.392+0000 7fd3a6a0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.392+0000 7fd3a6a0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.392+0000 7fd3a6a0e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.392+0000 7fd3a6a0e640 1 -- 172.21.15.7:0/758239465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a0067ac0 con 0x7fd3a00a8a30 2023-11-25T05:04:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.403+0000 7fe45b5e9640 10 monclient: get_monmap_and_config 2023-11-25T05:04:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.403+0000 7fe45b5e9640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.403+0000 7fe45b5e9640 10 monclient: monmap: 2023-11-25T05:04:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:36.404843+0000 2023-11-25T05:04:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:36.404843+0000 2023-11-25T05:04:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:36.409 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.404+0000 7fe45b5e9640 1 Processor -- start 2023-11-25T05:04:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.404+0000 7fe45b5e9640 1 -- start start 2023-11-25T05:04:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.404+0000 7fe45b5e9640 10 monclient: init 2023-11-25T05:04:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.405+0000 7fe45b5e9640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.405+0000 7fe45b5e9640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-11-25T05:04:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.405+0000 7fe45b5e9640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe4540a8a30 0x7fe4541424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.405+0000 7fe45b5e9640 10 monclient(hunting): picked mon.noname-c con 0x7fe4540a8a30 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.405+0000 7fe45b5e9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4540a8e10 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.405+0000 7fe45b5e9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe454142a20 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.406+0000 7fe45b5e9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe454142e70 con 0x7fe4540a8e10 2023-11-25T05:04:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.406+0000 7fe45b5e9640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4541462c0 con 0x7fe454142a20 2023-11-25T05:04:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.406+0000 7fe45b5e9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.406+0000 7fe45b5e9640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4540aaa10 con 0x7fe4540a8a30 2023-11-25T05:04:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.406+0000 7fe45b5e9640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.406+0000 7fe45b5e9640 10 monclient(hunting): authenticate will time out at 1402.648936s 2023-11-25T05:04:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.409+0000 7fd39cff9640 1 -- 172.21.15.7:0/758239465 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd390007490 con 0x7fd3a00a8a30 2023-11-25T05:04:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.409+0000 7fd39cff9640 1 --2- 172.21.15.7:0/758239465 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fd38c06c820 0x7fd38c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.409+0000 7fd39f7fe640 1 --2- 172.21.15.7:0/758239465 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fd38c06c820 0x7fd38c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.409+0000 7fd39f7fe640 10 monclient: get_auth_request con 0x7fd38c06c820 auth_method 0 2023-11-25T05:04:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.410+0000 7fd39cff9640 10 client.4339.objecter ms_handle_connect 0x7fd38c06c820 2023-11-25T05:04:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.410+0000 7fd39f7fe640 1 --2- 172.21.15.7:0/758239465 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fd38c06c820 0x7fd38c06ece0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0072c60 tx=0x7fd394006bf0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.411+0000 7fd39cff9640 1 -- 172.21.15.7:0/758239465 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd390057dc0 con 0x7fd3a00a8a30 2023-11-25T05:04:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.411+0000 7fd39cff9640 10 client.4339.objecter ms_dispatch 0x7fd3a0000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:36.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.411+0000 7fd39cff9640 3 client.4339.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:36.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.411+0000 7fd39cff9640 3 client.4339.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:36.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.411+0000 7fd39cff9640 20 client.4339.objecter dump_active .. 0 homeless 2023-11-25T05:04:36.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.412+0000 7fe459385640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe4540a8a30 0x7fe4541424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.413+0000 7fe459b86640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe454142a20 0x7fe454145bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:52626/0 (socket says 172.21.15.7:52626) 2023-11-25T05:04:36.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.413+0000 7fe459b86640 1 -- 172.21.15.7:0/4199716210 learned_addr learned my addr 172.21.15.7:0/4199716210 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:36.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.413+0000 7fe459385640 10 monclient(hunting): get_auth_request con 0x7fe4540a8a30 auth_method 0 2023-11-25T05:04:36.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.413+0000 7fe459385640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.413+0000 7fe459385640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.413+0000 7fe459385640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.428+0000 7f4f9add5640 10 monclient: get_monmap_and_config 2023-11-25T05:04:36.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.428+0000 7f4f9add5640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.428+0000 7f4f9add5640 10 monclient: monmap: 2023-11-25T05:04:36.431 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.431 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.431 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:36.429890+0000 2023-11-25T05:04:36.431 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:36.429890+0000 2023-11-25T05:04:36.431 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.431 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.431 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:36.431 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:36.431 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:36.431 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:36.431 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:36.432 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.429+0000 7f4f9add5640 1 Processor -- start 2023-11-25T05:04:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.429+0000 7f4f9add5640 1 -- start start 2023-11-25T05:04:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.429+0000 7f4f9add5640 10 monclient: init 2023-11-25T05:04:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.430+0000 7f4f9add5640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.430+0000 7f4f9add5640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2023-11-25T05:04:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.430+0000 7f4f9add5640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f940a8a30 0x7f4f941424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.430+0000 7f4f9add5640 10 monclient(hunting): picked mon.noname-b con 0x7f4f940a8a30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.430+0000 7f4f9add5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f940a8e10 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.430+0000 7f4f9add5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4f94142a20 0x7f4f94145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.430+0000 7f4f9add5640 10 monclient(hunting): picked mon.noname-a con 0x7f4f94142a20 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.430+0000 7f4f9add5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f94142eb0 con 0x7f4f940a8e10 2023-11-25T05:04:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.430+0000 7f4f9add5640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.430+0000 7f4f9add5640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f940aaa10 con 0x7f4f94142a20 2023-11-25T05:04:36.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.430+0000 7f4f9add5640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.430+0000 7f4f9add5640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f940aac10 con 0x7f4f940a8a30 2023-11-25T05:04:36.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.430+0000 7f4f9add5640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.430+0000 7f4f9add5640 10 monclient(hunting): authenticate will time out at 1402.673420s 2023-11-25T05:04:36.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.437+0000 7f5246a1f640 10 monclient: get_monmap_and_config 2023-11-25T05:04:36.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.437+0000 7f5246a1f640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.437+0000 7f5246a1f640 10 monclient: monmap: 2023-11-25T05:04:36.439 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.439 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.439 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:36.438509+0000 2023-11-25T05:04:36.440 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:36.438509+0000 2023-11-25T05:04:36.440 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.440 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.440 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:36.440 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:36.440 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:36.441 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:36.441 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:36.441 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.438+0000 7f5246a1f640 1 Processor -- start 2023-11-25T05:04:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.438+0000 7f5246a1f640 1 -- start start 2023-11-25T05:04:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.438+0000 7f5246a1f640 10 monclient: init 2023-11-25T05:04:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.439+0000 7f5246a1f640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.439+0000 7f5246a1f640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2023-11-25T05:04:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.439+0000 7f5246a1f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f52400a8a30 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.439+0000 7f5246a1f640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5240142bf0 0x7f524013ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.439+0000 7f5246a1f640 10 monclient(hunting): picked mon.noname-c con 0x7f5240142bf0 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.439+0000 7f5246a1f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52400a8e10 0x7f524013f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.439+0000 7f5246a1f640 10 monclient(hunting): picked mon.noname-b con 0x7f52400a8e10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.439+0000 7f5246a1f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52400aac10 con 0x7f52400a8a30 2023-11-25T05:04:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.439+0000 7f5246a1f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.439+0000 7f4f98b71640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f940a8a30 0x7f4f941424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.439+0000 7f5246a1f640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52400aaa10 con 0x7f5240142bf0 2023-11-25T05:04:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.439+0000 7f5246a1f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.439+0000 7f5246a1f640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52400aaef0 con 0x7f52400a8e10 2023-11-25T05:04:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.439+0000 7f5246a1f640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.439+0000 7f5246a1f640 10 monclient(hunting): authenticate will time out at 1402.682203s 2023-11-25T05:04:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.440+0000 7f4f98b71640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f940a8a30 0x7f4f941424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33726/0 (socket says 172.21.15.7:33726) 2023-11-25T05:04:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.440+0000 7f4f98b71640 1 -- 172.21.15.7:0/1445057115 learned_addr learned my addr 172.21.15.7:0/1445057115 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.440+0000 7f4f98b71640 10 monclient(hunting): get_auth_request con 0x7f4f940a8a30 auth_method 0 2023-11-25T05:04:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.440+0000 7f4f98b71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.440+0000 7f4f98b71640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.440+0000 7f4f98b71640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.463+0000 7f4f98b71640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.463+0000 7f4f98b71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.463+0000 7f4f98b71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.468+0000 7f4f98b71640 10 monclient(hunting): handle_auth_done global_id 4280 payload 274 2023-11-25T05:04:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.468+0000 7f4f98b71640 1 -- 172.21.15.7:0/1445057115 >> v2:172.21.15.7:3300/0 conn(0x7f4f94142a20 msgr2=0x7f4f94145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.468+0000 7f4f98b71640 1 --2- 172.21.15.7:0/1445057115 >> v2:172.21.15.7:3300/0 conn(0x7f4f94142a20 0x7f4f94145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.468+0000 7f4f98b71640 1 -- 172.21.15.7:0/1445057115 >> v1:172.21.15.7:6789/0 conn(0x7f4f940a8e10 legacy=0x7f4f941454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.468+0000 7f4f98b71640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.468+0000 7f4f98b71640 1 monclient: found mon.noname-b 2023-11-25T05:04:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.468+0000 7f4f98b71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.468+0000 7f4f98b71640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.468+0000 7f4f98b71640 1 -- 172.21.15.7:0/1445057115 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f940aaef0 con 0x7f4f940a8a30 2023-11-25T05:04:36.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7f4f98b71640 1 --2- 172.21.15.7:0/1445057115 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f940a8a30 0x7f4f941424e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4f840098a0 tx=0x7f4f8402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=debac649f578a458 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff73d59a640 1 -- 172.21.15.7:0/432146691 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73802cac0 con 0x7ff740143780 2023-11-25T05:04:36.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff73d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff73d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:36.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff73d59a640 10 monclient: dump: 2023-11-25T05:04:36.477 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.477 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.477 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.477 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.477 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.477 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.477 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.477 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.478 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.478 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff73d59a640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff73d59a640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff73d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff73d59a640 1 -- 172.21.15.7:0/432146691 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73802cc20 con 0x7ff740143780 2023-11-25T05:04:36.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff74487c640 5 monclient: authenticate success, global_id 4274 2023-11-25T05:04:36.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff74487c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:36.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff73d59a640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff74487c640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:36.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff74487c640 4 set_mon_vals no callback set 2023-11-25T05:04:36.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff73d59a640 1 -- 172.21.15.7:0/432146691 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7380343f0 con 0x7ff740143780 2023-11-25T05:04:36.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff73d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff73d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:36.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.474+0000 7ff73d59a640 10 monclient: dump: 2023-11-25T05:04:36.480 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.481 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.481 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.481 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.481 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.481 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.481 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.482 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.482 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.482 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.475+0000 7ff74487c640 10 monclient: shutdown 2023-11-25T05:04:36.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.475+0000 7ff74487c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.475+0000 7ff74487c640 1 -- 172.21.15.7:0/432146691 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff740143780 msgr2=0x7ff740150c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.475+0000 7ff74487c640 1 --2- 172.21.15.7:0/432146691 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff740143780 0x7ff740150c80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff738009e50 tx=0x7ff73802bf20 comp rx=0 tx=0).stop 2023-11-25T05:04:36.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.475+0000 7ff74487c640 1 -- 172.21.15.7:0/432146691 shutdown_connections 2023-11-25T05:04:36.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.475+0000 7ff74487c640 1 --2- 172.21.15.7:0/432146691 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff740143780 0x7ff740150c80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.475+0000 7ff74487c640 1 --2- 172.21.15.7:0/432146691 >> v2:172.21.15.7:3300/0 conn(0x7ff7400b29e0 0x7ff740143240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.475+0000 7ff74487c640 1 -- 172.21.15.7:0/432146691 >> 172.21.15.7:0/432146691 conn(0x7ff740061a30 msgr2=0x7ff7400b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.475+0000 7ff74487c640 1 -- 172.21.15.7:0/432146691 shutdown_connections 2023-11-25T05:04:36.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.475+0000 7ff74487c640 1 -- 172.21.15.7:0/432146691 wait complete. 2023-11-25T05:04:36.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.476+0000 7ff74487c640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.476+0000 7ff74487c640 10 monclient: monmap: 2023-11-25T05:04:36.485 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.485 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:36.485 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:36.486 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.476+0000 7ff74487c640 1 librados: starting msgr at 2023-11-25T05:04:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.476+0000 7ff74487c640 1 librados: starting objecter 2023-11-25T05:04:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.476+0000 7ff74487c640 1 Processor -- start 2023-11-25T05:04:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.476+0000 7ff74487c640 1 -- start start 2023-11-25T05:04:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.476+0000 7ff74487c640 1 librados: setting wanted keys 2023-11-25T05:04:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.476+0000 7ff74487c640 1 librados: calling monclient init 2023-11-25T05:04:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.476+0000 7ff74487c640 10 monclient: init 2023-11-25T05:04:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-25T05:04:36.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7400b0f20 0x7ff7400abec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 10 monclient(hunting): picked mon.noname-b con 0x7ff7400b0f20 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:36.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff7400b29e0 0x7ff740133ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 10 monclient(hunting): picked mon.noname-c con 0x7ff7400b29e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff740143780 0x7ff7400aca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 10 monclient(hunting): picked mon.noname-a con 0x7ff740143780 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:36.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7400ab0a0 con 0x7ff740143780 2023-11-25T05:04:36.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7400ae320 con 0x7ff7400b0f20 2023-11-25T05:04:36.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7400ac560 con 0x7ff7400b29e0 2023-11-25T05:04:36.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.477+0000 7ff74487c640 10 monclient(hunting): authenticate will time out at 1402.720165s 2023-11-25T05:04:36.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.479+0000 7f5244fbc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52400a8e10 0x7f524013f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.479+0000 7ff73dd9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff7400b29e0 0x7ff740133ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.480+0000 7ff73dd9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff7400b29e0 0x7ff740133ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33746/0 (socket says 172.21.15.7:33746) 2023-11-25T05:04:36.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.480+0000 7ff73dd9b640 1 -- 172.21.15.7:0/4102865996 learned_addr learned my addr 172.21.15.7:0/4102865996 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.480+0000 7ff73dd9b640 10 monclient(hunting): get_auth_request con 0x7ff7400b29e0 auth_method 0 2023-11-25T05:04:36.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.480+0000 7ff73dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.480+0000 7ff73dd9b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.480+0000 7ff73dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.480+0000 7f5244fbc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52400a8e10 0x7f524013f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33742/0 (socket says 172.21.15.7:33742) 2023-11-25T05:04:36.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.480+0000 7f5244fbc640 1 -- 172.21.15.7:0/2792510388 learned_addr learned my addr 172.21.15.7:0/2792510388 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.480+0000 7f5244fbc640 10 monclient(hunting): get_auth_request con 0x7f52400a8e10 auth_method 0 2023-11-25T05:04:36.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.480+0000 7f5244fbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.480+0000 7f5244fbc640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.480+0000 7f5244fbc640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23bffff640 1 -- 172.21.15.7:0/412701693 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c0009510 con 0x7f23c8142a20 2023-11-25T05:04:36.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23bffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:36.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23bffff640 10 monclient: dump: 2023-11-25T05:04:36.493 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.493 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.493 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.493 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.493 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.494 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.494 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.494 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.494 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.494 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23bffff640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23bffff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23bffff640 1 -- 172.21.15.7:0/412701693 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c000ee70 con 0x7f23c8142a20 2023-11-25T05:04:36.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23cd01f640 5 monclient: authenticate success, global_id 4262 2023-11-25T05:04:36.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23cd01f640 10 monclient: _renew_subs 2023-11-25T05:04:36.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23cd01f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23cd01f640 1 -- 172.21.15.7:0/412701693 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c80af920 con 0x7f23c8142a20 2023-11-25T05:04:36.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23bffff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23bffff640 1 -- 172.21.15.7:0/412701693 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c000cb80 con 0x7f23c8142a20 2023-11-25T05:04:36.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:36.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23bffff640 10 monclient: dump: 2023-11-25T05:04:36.495 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.495 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.495 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.495 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.496 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.496 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.496 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.496 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.497 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.497 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23c5d9b640 4 set_mon_vals no callback set 2023-11-25T05:04:36.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23cd01f640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23cd01f640 10 monclient: _renew_subs 2023-11-25T05:04:36.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23cd01f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23cd01f640 1 -- 172.21.15.7:0/412701693 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c813c320 con 0x7f23c8142a20 2023-11-25T05:04:36.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.481+0000 7f23cd01f640 1 librados: init done 2023-11-25T05:04:36.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.482+0000 7f5244fbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.482+0000 7f5244fbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.482+0000 7f5244fbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.482+0000 7f23cd01f640 10 client.4262.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.482+0000 7f23cd01f640 15 client.4262.objecter _session_command_op_assign -1 1 2023-11-25T05:04:36.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.482+0000 7f23cd01f640 20 client.4262.objecter _get_session osd=-1 returning homeless 2023-11-25T05:04:36.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.482+0000 7f23cd01f640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.482+0000 7f23cd01f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.482+0000 7f23cd01f640 1 -- 172.21.15.7:0/412701693 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f23c8141fc0 con 0x7f23c8142a20 2023-11-25T05:04:36.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.483+0000 7ff73dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.483+0000 7ff73dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.483+0000 7ff73dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f5244fbc640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2023-11-25T05:04:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f5244fbc640 1 -- 172.21.15.7:0/2792510388 >> v2:172.21.15.90:3300/0 conn(0x7f5240142bf0 msgr2=0x7f524013ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f5244fbc640 1 --2- 172.21.15.7:0/2792510388 >> v2:172.21.15.90:3300/0 conn(0x7f5240142bf0 0x7f524013ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f5244fbc640 1 -- 172.21.15.7:0/2792510388 >> v1:172.21.15.7:6789/0 conn(0x7f52400a8a30 legacy=0x7f52401424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f5244fbc640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f5244fbc640 1 monclient: found mon.noname-b 2023-11-25T05:04:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f5244fbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f5244fbc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f5244fbc640 1 -- 172.21.15.7:0/2792510388 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52400a97f0 con 0x7f52400a8e10 2023-11-25T05:04:36.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f937fe640 1 -- 172.21.15.7:0/1445057115 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84031640 con 0x7f4f940a8a30 2023-11-25T05:04:36.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:36.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f937fe640 10 monclient: dump: 2023-11-25T05:04:36.505 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.505 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.506 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.506 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.506 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.506 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.506 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.507 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.507 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.507 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f937fe640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f937fe640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f937fe640 1 -- 172.21.15.7:0/1445057115 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f840317a0 con 0x7f4f940a8a30 2023-11-25T05:04:36.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f937fe640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f937fe640 1 -- 172.21.15.7:0/1445057115 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84031b30 con 0x7f4f940a8a30 2023-11-25T05:04:36.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:36.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f937fe640 10 monclient: dump: 2023-11-25T05:04:36.509 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.509 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.509 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.509 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.509 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.510 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.510 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.510 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.514 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.515 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f9add5640 5 monclient: authenticate success, global_id 4280 2023-11-25T05:04:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f9add5640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.485+0000 7f4f9add5640 4 set_mon_vals no callback set 2023-11-25T05:04:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.486+0000 7f4f9add5640 10 monclient: shutdown 2023-11-25T05:04:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.486+0000 7f4f9add5640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.486+0000 7f4f9add5640 1 -- 172.21.15.7:0/1445057115 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f940a8a30 msgr2=0x7f4f941424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.486+0000 7f4f9add5640 1 --2- 172.21.15.7:0/1445057115 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f940a8a30 0x7f4f941424e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4f840098a0 tx=0x7f4f8402b8f0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.486+0000 7f4f9add5640 1 -- 172.21.15.7:0/1445057115 shutdown_connections 2023-11-25T05:04:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.486+0000 7f4f9add5640 1 --2- 172.21.15.7:0/1445057115 >> v2:172.21.15.7:3300/0 conn(0x7f4f94142a20 0x7f4f94145bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.486+0000 7f4f9add5640 1 --2- 172.21.15.7:0/1445057115 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f940a8a30 0x7f4f941424e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.486+0000 7f4f9add5640 1 -- 172.21.15.7:0/1445057115 >> 172.21.15.7:0/1445057115 conn(0x7f4f94061a30 msgr2=0x7f4f940b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.486+0000 7f4f9add5640 1 -- 172.21.15.7:0/1445057115 shutdown_connections 2023-11-25T05:04:36.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.487+0000 7f4f9add5640 1 -- 172.21.15.7:0/1445057115 wait complete. 2023-11-25T05:04:36.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.488+0000 7f4f9add5640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.488+0000 7f4f9add5640 10 monclient: monmap: 2023-11-25T05:04:36.516 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.516 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:36.516 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:36.516 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:36.516 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.516 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.516 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:36.517 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:36.517 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:36.517 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.488+0000 7f4f9add5640 1 librados: starting msgr at 2023-11-25T05:04:36.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.488+0000 7f4f9add5640 1 librados: starting objecter 2023-11-25T05:04:36.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.489+0000 7f4f9add5640 1 Processor -- start 2023-11-25T05:04:36.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.489+0000 7f4f9add5640 1 -- start start 2023-11-25T05:04:36.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.489+0000 7f4f9add5640 1 librados: setting wanted keys 2023-11-25T05:04:36.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.489+0000 7f4f9add5640 1 librados: calling monclient init 2023-11-25T05:04:36.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.489+0000 7f4f9add5640 10 monclient: init 2023-11-25T05:04:36.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.489+0000 7ff73dd9b640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2023-11-25T05:04:36.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.489+0000 7ff73dd9b640 1 -- 172.21.15.7:0/4102865996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7400b0f20 msgr2=0x7ff7400abec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.489+0000 7ff73dd9b640 1 --2- 172.21.15.7:0/4102865996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7400b0f20 0x7ff7400abec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.489+0000 7ff73dd9b640 1 -- 172.21.15.7:0/4102865996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff740143780 msgr2=0x7ff7400aca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.489+0000 7ff73dd9b640 1 --2- 172.21.15.7:0/4102865996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff740143780 0x7ff7400aca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.489+0000 7ff73dd9b640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.489+0000 7ff73dd9b640 1 monclient: found mon.noname-c 2023-11-25T05:04:36.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.489+0000 7ff73dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.489+0000 7ff73dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.489+0000 7ff73dd9b640 1 -- 172.21.15.7:0/4102865996 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff738009b30 con 0x7ff7400b29e0 2023-11-25T05:04:36.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:04:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f940a8a30 0x7f4f940aed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 10 monclient(hunting): picked mon.noname-c con 0x7f4f940a8a30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f940a8e10 0x7f4f940af930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 10 monclient(hunting): picked mon.noname-b con 0x7f4f940a8e10 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f94142a20 0x7f4f9413b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 10 monclient(hunting): picked mon.noname-a con 0x7f4f94142a20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f940b1220 con 0x7f4f94142a20 2023-11-25T05:04:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f940adf10 con 0x7f4f940a8e10 2023-11-25T05:04:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f940affa0 con 0x7f4f940a8a30 2023-11-25T05:04:36.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.490+0000 7f4f9add5640 10 monclient(hunting): authenticate will time out at 1402.733684s 2023-11-25T05:04:36.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.491+0000 7f5244fbc640 1 --2- 172.21.15.7:0/2792510388 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52400a8e10 0x7f524013f230 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5238009e50 tx=0x7f523802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e9663f36ca03891 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.509+0000 7ff73dd9b640 1 --2- 172.21.15.7:0/4102865996 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff7400b29e0 0x7ff740133ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff73400b4a0 tx=0x7ff73400b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.509+0000 7ff72effd640 10 client.?.objecter ms_handle_connect 0x7ff7400b29e0 2023-11-25T05:04:36.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.509+0000 7ff72effd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:36.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.509+0000 7f77b576b640 1 --2- 172.21.15.7:0/4120082866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a80c8a30 0x7f77a80d3610 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f77b0063e60 tx=0x7f77b00a5d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dffea85e0fc87d12 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.523+0000 7f23bffff640 1 -- 172.21.15.7:0/412701693 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f23c000cce0 con 0x7f23c8142a20 2023-11-25T05:04:36.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.523+0000 7f23bffff640 1 --2- 172.21.15.7:0/412701693 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f23a806c820 0x7f23a806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.523+0000 7f23c6d9d640 1 --2- 172.21.15.7:0/412701693 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f23a806c820 0x7f23a806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.523+0000 7f23c6d9d640 10 monclient: get_auth_request con 0x7f23a806c820 auth_method 0 2023-11-25T05:04:36.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.524+0000 7f23bffff640 10 client.4262.objecter ms_handle_connect 0x7f23a806c820 2023-11-25T05:04:36.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.524+0000 7f23c6d9d640 1 --2- 172.21.15.7:0/412701693 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f23a806c820 0x7f23a806ece0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f23b8035000 tx=0x7f23b8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 1 -- 172.21.15.7:0/412701693 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f23c0058b80 con 0x7f23c8142a20 2023-11-25T05:04:36.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 10 client.4262.objecter ms_dispatch 0x7f23c8000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:36.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 3 client.4262.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:36.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 3 client.4262.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:36.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 10 client.4262.objecter checking command 1 2023-11-25T05:04:36.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 1 --2- 172.21.15.7:0/412701693 >> [v2:172.21.15.90:6818/3560048676,v1:172.21.15.90:6819/3560048676] conn(0x7f23a8072c00 0x7f23a8075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f4f98b71640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f940a8a30 0x7f4f940aed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 20 client.4262.objecter _get_session s=0x7f23a8072530 osd=6 3 2023-11-25T05:04:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 20 client.4262.objecter put_session s=0x7f23a8072530 osd=6 3 2023-11-25T05:04:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 15 client.4262.objecter _session_command_op_remove -1 1 2023-11-25T05:04:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 20 client.4262.objecter _get_session s=0x7f23a8072530 osd=6 3 2023-11-25T05:04:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 20 client.4262.objecter get_session s=0x7f23a8072530 osd=6 3 2023-11-25T05:04:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 15 client.4262.objecter _session_command_op_assign 6 1 2023-11-25T05:04:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 20 client.4262.objecter put_session s=0x7f23a8072530 osd=6 4 2023-11-25T05:04:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 10 client.4262.objecter _send_command 1 2023-11-25T05:04:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 1 -- 172.21.15.7:0/412701693 --> [v2:172.21.15.90:6818/3560048676,v1:172.21.15.90:6819/3560048676] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f23a80756d0 con 0x7f23a8072c00 2023-11-25T05:04:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23bffff640 20 client.4262.objecter dump_active .. 0 homeless 2023-11-25T05:04:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.526+0000 7f23be59c640 1 --2- 172.21.15.7:0/412701693 >> [v2:172.21.15.90:6818/3560048676,v1:172.21.15.90:6819/3560048676] conn(0x7f23a8072c00 0x7f23a8075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.527+0000 7f23be59c640 10 monclient: get_auth_request con 0x7f23a8072c00 auth_method 0 2023-11-25T05:04:36.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.528+0000 7f4f98b71640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f940a8a30 0x7f4f940aed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33756/0 (socket says 172.21.15.7:33756) 2023-11-25T05:04:36.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.528+0000 7f4f98b71640 1 -- 172.21.15.7:0/3084147837 learned_addr learned my addr 172.21.15.7:0/3084147837 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.528+0000 7f4f98b71640 10 monclient(hunting): get_auth_request con 0x7f4f940a8a30 auth_method 0 2023-11-25T05:04:36.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.528+0000 7f4f98b71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.528+0000 7f4f98b71640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.528+0000 7f4f98b71640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.529+0000 7f4f98b71640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.529+0000 7f4f98b71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.529+0000 7f4f98b71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.530+0000 7f23bffff640 10 client.4262.objecter ms_handle_connect 0x7f23a8072c00 2023-11-25T05:04:36.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.530+0000 7f23be59c640 1 --2- 172.21.15.7:0/412701693 >> [v2:172.21.15.90:6818/3560048676,v1:172.21.15.90:6819/3560048676] conn(0x7f23a8072c00 0x7f23a8075020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.530+0000 7f523effd640 1 -- 172.21.15.7:0/2792510388 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523802ca70 con 0x7f52400a8e10 2023-11-25T05:04:36.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.530+0000 7f523effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f523effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:36.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f523effd640 10 monclient: dump: 2023-11-25T05:04:36.533 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.533 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.533 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.533 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.533 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.533 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.533 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.533 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.534 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.534 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f523effd640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f523effd640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f523effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f523effd640 1 -- 172.21.15.7:0/2792510388 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523802cbd0 con 0x7f52400a8e10 2023-11-25T05:04:36.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f523effd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f523effd640 1 -- 172.21.15.7:0/2792510388 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52380343f0 con 0x7f52400a8e10 2023-11-25T05:04:36.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f5246a1f640 5 monclient: authenticate success, global_id 4283 2023-11-25T05:04:36.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f5246a1f640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:36.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f5246a1f640 4 set_mon_vals no callback set 2023-11-25T05:04:36.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f523effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f523effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:36.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f523effd640 10 monclient: dump: 2023-11-25T05:04:36.535 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.535 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.535 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.535 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.535 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.535 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.535 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.535 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.535 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.535 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f5246a1f640 10 monclient: shutdown 2023-11-25T05:04:36.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f5246a1f640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f5246a1f640 1 -- 172.21.15.7:0/2792510388 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52400a8e10 msgr2=0x7f524013f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.531+0000 7f5246a1f640 1 --2- 172.21.15.7:0/2792510388 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52400a8e10 0x7f524013f230 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5238009e50 tx=0x7f523802bed0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.532+0000 7f5246a1f640 1 -- 172.21.15.7:0/2792510388 shutdown_connections 2023-11-25T05:04:36.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.532+0000 7f5246a1f640 1 --2- 172.21.15.7:0/2792510388 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52400a8e10 0x7f524013f230 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.532+0000 7f5246a1f640 1 --2- 172.21.15.7:0/2792510388 >> v2:172.21.15.90:3300/0 conn(0x7f5240142bf0 0x7f524013ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.532+0000 7f5246a1f640 1 -- 172.21.15.7:0/2792510388 >> 172.21.15.7:0/2792510388 conn(0x7f5240061a30 msgr2=0x7f52400b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.532+0000 7f5246a1f640 1 -- 172.21.15.7:0/2792510388 shutdown_connections 2023-11-25T05:04:36.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.532+0000 7f5246a1f640 1 -- 172.21.15.7:0/2792510388 wait complete. 2023-11-25T05:04:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.533+0000 7f5246a1f640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.533+0000 7f5246a1f640 10 monclient: monmap: 2023-11-25T05:04:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:36.539 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.533+0000 7f5246a1f640 1 librados: starting msgr at 2023-11-25T05:04:36.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.533+0000 7f5246a1f640 1 librados: starting objecter 2023-11-25T05:04:36.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.533+0000 7f5246a1f640 1 Processor -- start 2023-11-25T05:04:36.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.533+0000 7f5246a1f640 1 -- start start 2023-11-25T05:04:36.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.533+0000 7f5246a1f640 1 librados: setting wanted keys 2023-11-25T05:04:36.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.533+0000 7f5246a1f640 1 librados: calling monclient init 2023-11-25T05:04:36.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.533+0000 7f5246a1f640 10 monclient: init 2023-11-25T05:04:36.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.534+0000 7f5246a1f640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.534+0000 7f5246a1f640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:04:36.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.534+0000 7f5246a1f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52400a8a30 0x7f52400b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.534+0000 7f5246a1f640 10 monclient(hunting): picked mon.noname-c con 0x7f52400a8a30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.534+0000 7f5246a1f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f52400a8e10 0x7f52400ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.534+0000 7f5246a1f640 10 monclient(hunting): picked mon.noname-b con 0x7f52400a8e10 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:36.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.534+0000 7f5246a1f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5240142bf0 0x7f52401da960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.534+0000 7f5246a1f640 10 monclient(hunting): picked mon.noname-a con 0x7f5240142bf0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:36.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.534+0000 7f5246a1f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.534+0000 7f5246a1f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52401da440 con 0x7f5240142bf0 2023-11-25T05:04:36.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.534+0000 7f5246a1f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.534+0000 7f5246a1f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52401daea0 con 0x7f52400a8e10 2023-11-25T05:04:36.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.534+0000 7f5246a1f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.534+0000 7f5246a1f640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52401db060 con 0x7f52400a8a30 2023-11-25T05:04:36.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.535+0000 7f5246a1f640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.535+0000 7f5246a1f640 10 monclient(hunting): authenticate will time out at 1402.777866s 2023-11-25T05:04:36.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.535+0000 7f4f98b71640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2023-11-25T05:04:36.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.535+0000 7f4f98b71640 1 -- 172.21.15.7:0/3084147837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f940a8e10 msgr2=0x7f4f940af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.535+0000 7f4f98b71640 1 --2- 172.21.15.7:0/3084147837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f940a8e10 0x7f4f940af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.535+0000 7f4f98b71640 1 -- 172.21.15.7:0/3084147837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f94142a20 msgr2=0x7f4f9413b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.535+0000 7f4f98b71640 1 --2- 172.21.15.7:0/3084147837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f94142a20 0x7f4f9413b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.535+0000 7f4f98b71640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.535+0000 7f4f98b71640 1 monclient: found mon.noname-c 2023-11-25T05:04:36.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.535+0000 7f4f98b71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.535+0000 7f4f98b71640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.535+0000 7f4f98b71640 1 -- 172.21.15.7:0/3084147837 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f840094d0 con 0x7f4f940a8a30 2023-11-25T05:04:36.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.539+0000 7ff72effd640 1 -- 172.21.15.7:0/4102865996 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff734011830 con 0x7ff7400b29e0 2023-11-25T05:04:36.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.539+0000 7ff72effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.539+0000 7ff72effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:36.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.539+0000 7ff72effd640 10 monclient: dump: 2023-11-25T05:04:36.544 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.544 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.544 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.544 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.544 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.544 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.544 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.544 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.544 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.539+0000 7ff72effd640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.539+0000 7ff72effd640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.539+0000 7ff72effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.539+0000 7ff72effd640 1 -- 172.21.15.7:0/4102865996 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff734011990 con 0x7ff7400b29e0 2023-11-25T05:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.539+0000 7ff74487c640 5 monclient: authenticate success, global_id 4286 2023-11-25T05:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.539+0000 7ff74487c640 10 monclient: _renew_subs 2023-11-25T05:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7ff74487c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7ff74487c640 1 -- 172.21.15.7:0/4102865996 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff740147290 con 0x7ff7400b29e0 2023-11-25T05:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7ff72effd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7ff72effd640 1 -- 172.21.15.7:0/4102865996 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff734011d40 con 0x7ff7400b29e0 2023-11-25T05:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7ff72effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7ff72effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:36.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7ff72effd640 10 monclient: dump: 2023-11-25T05:04:36.546 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.547 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.547 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.550 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.550 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.550 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7ff74487c640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7ff74487c640 10 monclient: _renew_subs 2023-11-25T05:04:36.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7ff74487c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7ff74487c640 1 -- 172.21.15.7:0/4102865996 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7400acfc0 con 0x7ff7400b29e0 2023-11-25T05:04:36.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7ff74487c640 1 librados: init done 2023-11-25T05:04:36.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7ff73cd99640 4 set_mon_vals no callback set 2023-11-25T05:04:36.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7f4f98b71640 1 --2- 172.21.15.7:0/3084147837 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f940a8a30 0x7f4f940aed50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4f8402b930 tx=0x7f4f840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7f4f917fa640 10 client.?.objecter ms_handle_connect 0x7f4f940a8a30 2023-11-25T05:04:36.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.540+0000 7f4f917fa640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:36.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.541+0000 7ff74487c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.541+0000 7ff74487c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.541+0000 7ff74487c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.541+0000 7ff74487c640 1 -- 172.21.15.7:0/4102865996 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff740067ac0 con 0x7ff7400b29e0 2023-11-25T05:04:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.576+0000 7f23bffff640 1 -- 172.21.15.7:0/412701693 <== osd.6 v2:172.21.15.90:6818/3560048676 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f23a80756d0 con 0x7f23a8072c00 2023-11-25T05:04:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.576+0000 7f23bffff640 10 client.4262.objecter ms_dispatch 0x7f23c8000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:04:36.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.576+0000 7f23bffff640 10 client.4262.objecter _finish_command 1 = system:0 2023-11-25T05:04:36.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.576+0000 7f23bffff640 20 client.4262.objecter put_session s=0x7f23a8072530 osd=6 4 2023-11-25T05:04:36.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.576+0000 7f23bffff640 15 client.4262.objecter _session_command_op_remove 6 1 2023-11-25T05:04:36.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.590+0000 7f23cd01f640 10 client.4262.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-25T05:04:36.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.590+0000 7f23cd01f640 15 client.4262.objecter _session_command_op_assign -1 2 2023-11-25T05:04:36.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.590+0000 7f23cd01f640 20 client.4262.objecter _get_session s=0x7f23a8072530 osd=6 3 2023-11-25T05:04:36.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.590+0000 7f23cd01f640 20 client.4262.objecter put_session s=0x7f23a8072530 osd=6 3 2023-11-25T05:04:36.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.590+0000 7f23cd01f640 20 client.4262.objecter _get_session s=0x7f23a8072530 osd=6 3 2023-11-25T05:04:36.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.590+0000 7f23cd01f640 15 client.4262.objecter _session_command_op_remove -1 2 2023-11-25T05:04:36.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.590+0000 7f23cd01f640 20 client.4262.objecter get_session s=0x7f23a8072530 osd=6 3 2023-11-25T05:04:36.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.590+0000 7f23cd01f640 15 client.4262.objecter _session_command_op_assign 6 2 2023-11-25T05:04:36.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.590+0000 7f23cd01f640 20 client.4262.objecter put_session s=0x7f23a8072530 osd=6 4 2023-11-25T05:04:36.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.590+0000 7f23cd01f640 10 client.4262.objecter _send_command 2 2023-11-25T05:04:36.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.590+0000 7f23cd01f640 1 -- 172.21.15.7:0/412701693 --> [v2:172.21.15.90:6818/3560048676,v1:172.21.15.90:6819/3560048676] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f23c8067ac0 con 0x7f23a8072c00 2023-11-25T05:04:36.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23bffff640 1 -- 172.21.15.7:0/412701693 <== osd.6 v2:172.21.15.90:6818/3560048676 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f23c8067ac0 con 0x7f23a8072c00 2023-11-25T05:04:36.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23bffff640 10 client.4262.objecter ms_dispatch 0x7f23c8000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:04:36.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23bffff640 10 client.4262.objecter _finish_command 2 = system:0 2023-11-25T05:04:36.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23bffff640 20 client.4262.objecter put_session s=0x7f23a8072530 osd=6 4 2023-11-25T05:04:36.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23bffff640 15 client.4262.objecter _session_command_op_remove 6 2 2023-11-25T05:04:36.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 10 librados: watch_flush enter 2023-11-25T05:04:36.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 10 librados: watch_flush exit 2023-11-25T05:04:36.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 10 client.4262.objecter close_session for osd.6 2023-11-25T05:04:36.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 1 -- 172.21.15.7:0/412701693 >> [v2:172.21.15.90:6818/3560048676,v1:172.21.15.90:6819/3560048676] conn(0x7f23a8072c00 msgr2=0x7f23a8075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 1 --2- 172.21.15.7:0/412701693 >> [v2:172.21.15.90:6818/3560048676,v1:172.21.15.90:6819/3560048676] conn(0x7f23a8072c00 0x7f23a8075020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 20 client.4262.objecter put_session s=0x7f23a8072530 osd=6 1 2023-11-25T05:04:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 20 client.4262.objecter shutdown clearing up homeless session... 2023-11-25T05:04:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 10 client.4262.objecter successfully canceled tick 2023-11-25T05:04:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 1 -- 172.21.15.7:0/412701693 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f23a806c820 msgr2=0x7f23a806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 1 --2- 172.21.15.7:0/412701693 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f23a806c820 0x7f23a806ece0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f23b8035000 tx=0x7f23b8030000 comp rx=0 tx=0).stop 2023-11-25T05:04:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 10 monclient: shutdown 2023-11-25T05:04:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 20 monclient: shutdown canceling and discarding version request 1 2023-11-25T05:04:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 1 -- 172.21.15.7:0/412701693 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23c8142a20 msgr2=0x7f23c813bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 1 --2- 172.21.15.7:0/412701693 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23c8142a20 0x7f23c813bde0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f23c000e820 tx=0x7f23c000ec20 comp rx=0 tx=0).stop 2023-11-25T05:04:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23c759e640 1 -- 172.21.15.7:0/412701693 reap_dead start 2023-11-25T05:04:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 1 -- 172.21.15.7:0/412701693 shutdown_connections 2023-11-25T05:04:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.596+0000 7f23cd01f640 1 -- 172.21.15.7:0/412701693 >> 172.21.15.7:0/412701693 conn(0x7f23c8061a30 msgr2=0x7f23c80b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.597+0000 7f23cd01f640 1 -- 172.21.15.7:0/412701693 shutdown_connections 2023-11-25T05:04:36.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.597+0000 7f23cd01f640 1 -- 172.21.15.7:0/412701693 wait complete. 2023-11-25T05:04:36.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.597+0000 7f23cd01f640 1 librados: shutdown 2023-11-25T05:04:36.614 INFO:teuthology.orchestra.run.smithi007.stdout:64424509446 2023-11-25T05:04:36.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-25T05:04:36.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.648+0000 7fd39cff9640 1 -- 172.21.15.7:0/758239465 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd390054100 con 0x7fd3a00a8a30 2023-11-25T05:04:36.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.648+0000 7fd39cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.648+0000 7fd39cff9640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:36.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.648+0000 7fe459385640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.648+0000 7fe459385640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.648+0000 7fe459385640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.652+0000 7fe459385640 10 monclient(hunting): handle_auth_done global_id 4342 payload 274 2023-11-25T05:04:36.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.652+0000 7fe459385640 1 -- 172.21.15.7:0/4199716210 >> v1:172.21.15.90:6789/0 conn(0x7fe454142a20 legacy=0x7fe454145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.652+0000 7fe459385640 1 -- 172.21.15.7:0/4199716210 >> v1:172.21.15.7:6789/0 conn(0x7fe4540a8e10 legacy=0x7fe4541454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:36.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.652+0000 7fe459385640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.652+0000 7fe459385640 1 monclient: found mon.noname-c 2023-11-25T05:04:36.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.652+0000 7fe459385640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.652+0000 7fe459385640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.652+0000 7fe459385640 1 -- 172.21.15.7:0/4199716210 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4540aac10 con 0x7fe4540a8a30 2023-11-25T05:04:36.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.657+0000 7fe459385640 1 --2- 172.21.15.7:0/4199716210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4540a8a30 0x7fe4541424e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe4500097a0 tx=0x7fe45002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ab0ae124026364a server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.660+0000 7f523f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f52400a8e10 0x7f52400ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.663+0000 7f523f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f52400a8e10 0x7f52400ad9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:35144/0 (socket says 172.21.15.7:35144) 2023-11-25T05:04:36.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.663+0000 7f523f7fe640 1 -- 172.21.15.7:0/790342916 learned_addr learned my addr 172.21.15.7:0/790342916 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.663+0000 7f523f7fe640 10 monclient(hunting): get_auth_request con 0x7f52400a8e10 auth_method 0 2023-11-25T05:04:36.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.663+0000 7f523f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.663+0000 7f523f7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.663+0000 7f523f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.664+0000 7f523f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.664+0000 7f523f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.664+0000 7f523f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.668+0000 7f523f7fe640 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2023-11-25T05:04:36.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.668+0000 7f523f7fe640 1 -- 172.21.15.7:0/790342916 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52400a8a30 msgr2=0x7f52400b0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.668+0000 7f523f7fe640 1 --2- 172.21.15.7:0/790342916 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52400a8a30 0x7f52400b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.668+0000 7f523f7fe640 1 -- 172.21.15.7:0/790342916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5240142bf0 msgr2=0x7f52401da960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.668+0000 7f523f7fe640 1 --2- 172.21.15.7:0/790342916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5240142bf0 0x7f52401da960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.668+0000 7f523f7fe640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.668+0000 7f523f7fe640 1 monclient: found mon.noname-b 2023-11-25T05:04:36.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.668+0000 7f523f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.668+0000 7f523f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.668+0000 7f523f7fe640 1 -- 172.21.15.7:0/790342916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5238009b30 con 0x7f52400a8e10 2023-11-25T05:04:36.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.670+0000 7f523f7fe640 1 --2- 172.21.15.7:0/790342916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f52400a8e10 0x7f52400ad9c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f523400b410 tx=0x7f523400b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.670+0000 7f523cff9640 10 client.?.objecter ms_handle_connect 0x7f52400a8e10 2023-11-25T05:04:36.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.671+0000 7f523cff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:36.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.674+0000 7fe44bfff640 1 -- 172.21.15.7:0/4199716210 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe450030610 con 0x7fe4540a8a30 2023-11-25T05:04:36.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.674+0000 7fe44bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.674+0000 7fe44bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.674+0000 7fe44bfff640 10 monclient: dump: 2023-11-25T05:04:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.677 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.674+0000 7fe44bfff640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.674+0000 7fe44bfff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.674+0000 7fe44bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.674+0000 7fe44bfff640 1 -- 172.21.15.7:0/4199716210 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe450030770 con 0x7fe4540a8a30 2023-11-25T05:04:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.674+0000 7fe45b5e9640 5 monclient: authenticate success, global_id 4342 2023-11-25T05:04:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.674+0000 7fe45b5e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.674+0000 7fe44bfff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.674+0000 7fe45b5e9640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.674+0000 7fe45b5e9640 4 set_mon_vals no callback set 2023-11-25T05:04:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.674+0000 7fe44bfff640 1 -- 172.21.15.7:0/4199716210 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe450030b00 con 0x7fe4540a8a30 2023-11-25T05:04:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7fe44bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7fe44bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7fe44bfff640 10 monclient: dump: 2023-11-25T05:04:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.679 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7f523cff9640 1 -- 172.21.15.7:0/790342916 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5234011830 con 0x7f52400a8e10 2023-11-25T05:04:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7f523cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7f523cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7f523cff9640 10 monclient: dump: 2023-11-25T05:04:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.681 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7f523cff9640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7f523cff9640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7f523cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7f523cff9640 1 -- 172.21.15.7:0/790342916 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5234011990 con 0x7f52400a8e10 2023-11-25T05:04:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7f523cff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7fe45b5e9640 10 monclient: shutdown 2023-11-25T05:04:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7fe45b5e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7fe45b5e9640 1 -- 172.21.15.7:0/4199716210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4540a8a30 msgr2=0x7fe4541424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7fe45b5e9640 1 --2- 172.21.15.7:0/4199716210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4540a8a30 0x7fe4541424e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe4500097a0 tx=0x7fe45002b960 comp rx=0 tx=0).stop 2023-11-25T05:04:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.675+0000 7f523e7fc640 4 set_mon_vals no callback set 2023-11-25T05:04:36.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7fe45b5e9640 1 -- 172.21.15.7:0/4199716210 shutdown_connections 2023-11-25T05:04:36.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7fe45b5e9640 1 --2- 172.21.15.7:0/4199716210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4540a8a30 0x7fe4541424e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7fe45b5e9640 1 -- 172.21.15.7:0/4199716210 >> 172.21.15.7:0/4199716210 conn(0x7fe454061a30 msgr2=0x7fe4540b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7fe45b5e9640 1 -- 172.21.15.7:0/4199716210 shutdown_connections 2023-11-25T05:04:36.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7fe45b5e9640 1 -- 172.21.15.7:0/4199716210 wait complete. 2023-11-25T05:04:36.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7f523cff9640 1 -- 172.21.15.7:0/790342916 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5234011d40 con 0x7f52400a8e10 2023-11-25T05:04:36.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7f523cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7f523cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:36.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7f523cff9640 10 monclient: dump: 2023-11-25T05:04:36.685 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.685 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.685 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.685 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.685 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.686 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7f5246a1f640 5 monclient: authenticate success, global_id 4345 2023-11-25T05:04:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7f5246a1f640 10 monclient: _renew_subs 2023-11-25T05:04:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7f5246a1f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7f5246a1f640 1 -- 172.21.15.7:0/790342916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52400ae480 con 0x7f52400a8e10 2023-11-25T05:04:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7f5246a1f640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7f5246a1f640 10 monclient: _renew_subs 2023-11-25T05:04:36.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7f5246a1f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7f5246a1f640 1 -- 172.21.15.7:0/790342916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52400ae8e0 con 0x7f52400a8e10 2023-11-25T05:04:36.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.676+0000 7f5246a1f640 1 librados: init done 2023-11-25T05:04:36.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.677+0000 7fe45b5e9640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.677+0000 7fe45b5e9640 10 monclient: monmap: 2023-11-25T05:04:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:36.689 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.678+0000 7f5246a1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.678+0000 7f5246a1f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.678+0000 7f5246a1f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.678+0000 7f5246a1f640 1 -- 172.21.15.7:0/790342916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5240067ac0 con 0x7f52400a8e10 2023-11-25T05:04:36.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.678+0000 7fe45b5e9640 1 librados: starting msgr at 2023-11-25T05:04:36.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.678+0000 7fe45b5e9640 1 librados: starting objecter 2023-11-25T05:04:36.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.678+0000 7fe45b5e9640 1 Processor -- start 2023-11-25T05:04:36.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.678+0000 7fe45b5e9640 1 -- start start 2023-11-25T05:04:36.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.678+0000 7fe45b5e9640 1 librados: setting wanted keys 2023-11-25T05:04:36.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.678+0000 7fe45b5e9640 1 librados: calling monclient init 2023-11-25T05:04:36.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.678+0000 7fe45b5e9640 10 monclient: init 2023-11-25T05:04:36.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:04:36.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe4540a8a30 0x7fe4541dab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 10 monclient(hunting): picked mon.noname-c con 0x7fe4540a8a30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4540a8e10 0x7fe45413acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 10 monclient(hunting): picked mon.noname-b con 0x7fe4540a8e10 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:36.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe454142a20 0x7fe45413b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 10 monclient(hunting): picked mon.noname-a con 0x7fe454142a20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:36.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45413b730 con 0x7fe454142a20 2023-11-25T05:04:36.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4540b0e20 con 0x7fe4540a8e10 2023-11-25T05:04:36.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4540b1100 con 0x7fe4540a8a30 2023-11-25T05:04:36.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.679+0000 7fe45b5e9640 10 monclient(hunting): authenticate will time out at 1402.922483s 2023-11-25T05:04:36.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.685+0000 7fe458b84640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4540a8e10 0x7fe45413acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.686+0000 7f2683fff640 1 -- 172.21.15.7:0/3597088794 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2690054080 con 0x7f269c0a8a30 2023-11-25T05:04:36.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.686+0000 7f2683fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.686+0000 7f2683fff640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:36.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.700+0000 7f523cff9640 1 -- 172.21.15.7:0/790342916 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f523401a400 con 0x7f52400a8e10 2023-11-25T05:04:36.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.701+0000 7f523cff9640 1 --2- 172.21.15.7:0/790342916 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f522c06c820 0x7f522c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.701+0000 7f523ffff640 1 --2- 172.21.15.7:0/790342916 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f522c06c820 0x7f522c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.701+0000 7f523ffff640 10 monclient: get_auth_request con 0x7f522c06c820 auth_method 0 2023-11-25T05:04:36.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.701+0000 7f523cff9640 10 client.4345.objecter ms_handle_connect 0x7f522c06c820 2023-11-25T05:04:36.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.701+0000 7f523ffff640 1 --2- 172.21.15.7:0/790342916 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f522c06c820 0x7f522c06ece0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5230001560 tx=0x7f5230007990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.707+0000 7fe459b86640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe454142a20 0x7fe45413b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.708+0000 7fe459b86640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe454142a20 0x7fe45413b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58284/0 (socket says 172.21.15.7:58284) 2023-11-25T05:04:36.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.708+0000 7fe459b86640 1 -- 172.21.15.7:0/2066593397 learned_addr learned my addr 172.21.15.7:0/2066593397 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.708+0000 7fe459b86640 10 monclient(hunting): get_auth_request con 0x7fe454142a20 auth_method 0 2023-11-25T05:04:36.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.708+0000 7fe459b86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.708+0000 7fe459b86640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.708+0000 7fe459b86640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.722+0000 7fe459b86640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.722+0000 7fe459b86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.722+0000 7fe459b86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.729+0000 7fe459b86640 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2023-11-25T05:04:36.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.729+0000 7fe459b86640 1 -- 172.21.15.7:0/2066593397 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe4540a8a30 msgr2=0x7fe4541dab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.729+0000 7fe459b86640 1 --2- 172.21.15.7:0/2066593397 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe4540a8a30 0x7fe4541dab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.729+0000 7fe459b86640 1 -- 172.21.15.7:0/2066593397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4540a8e10 msgr2=0x7fe45413acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.729+0000 7fe459b86640 1 --2- 172.21.15.7:0/2066593397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4540a8e10 0x7fe45413acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.729+0000 7fe459b86640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.729+0000 7fe459b86640 1 monclient: found mon.noname-a 2023-11-25T05:04:36.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.729+0000 7fe459b86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.729+0000 7fe459b86640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.729+0000 7fe459b86640 1 -- 172.21.15.7:0/2066593397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe450009400 con 0x7fe454142a20 2023-11-25T05:04:36.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.747+0000 7fe459b86640 1 --2- 172.21.15.7:0/2066593397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe454142a20 0x7fe45413b1f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00c9f0 tx=0x7fe44c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.747+0000 7fe449ffb640 10 client.?.objecter ms_handle_connect 0x7fe454142a20 2023-11-25T05:04:36.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.748+0000 7fe449ffb640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:36.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.799+0000 7fe5c6015640 10 monclient: get_monmap_and_config 2023-11-25T05:04:36.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.799+0000 7fe5c6015640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.799+0000 7fe5c6015640 10 monclient: monmap: 2023-11-25T05:04:36.801 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.801 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:36.800857+0000 2023-11-25T05:04:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:36.800857+0000 2023-11-25T05:04:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:36.802 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.801+0000 7fe5c6015640 1 Processor -- start 2023-11-25T05:04:36.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.801+0000 7fe5c6015640 1 -- start start 2023-11-25T05:04:36.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.801+0000 7fe5c6015640 10 monclient: init 2023-11-25T05:04:36.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.802+0000 7fe5c6015640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.802+0000 7fe5c6015640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2023-11-25T05:04:36.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.802+0000 7fe5c6015640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5c00a8a30 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:36.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.802+0000 7fe5c6015640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5c0142bf0 0x7fe5c013ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.802+0000 7fe5c6015640 10 monclient(hunting): picked mon.noname-b con 0x7fe5c0142bf0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.802+0000 7fe5c6015640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5c00a8e10 0x7fe5c013f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.802+0000 7fe5c6015640 10 monclient(hunting): picked mon.noname-a con 0x7fe5c00a8e10 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:36.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.802+0000 7fe5c6015640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c00aac10 con 0x7fe5c00a8a30 2023-11-25T05:04:36.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.802+0000 7fe5c6015640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.802+0000 7fe5c6015640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c00aaa10 con 0x7fe5c00a8e10 2023-11-25T05:04:36.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.802+0000 7fe5c6015640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.802+0000 7fe5c6015640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c00aaef0 con 0x7fe5c0142bf0 2023-11-25T05:04:36.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.802+0000 7fe5c6015640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.802+0000 7fe5c6015640 10 monclient(hunting): authenticate will time out at 1403.045184s 2023-11-25T05:04:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.815+0000 7fe5bffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5c00a8e10 0x7fe5c013f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.819+0000 7fe5bffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5c00a8e10 0x7fe5c013f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58292/0 (socket says 172.21.15.7:58292) 2023-11-25T05:04:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.819+0000 7fe5bffff640 1 -- 172.21.15.7:0/3596280263 learned_addr learned my addr 172.21.15.7:0/3596280263 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.819+0000 7fe5bffff640 10 monclient(hunting): get_auth_request con 0x7fe5c00a8e10 auth_method 0 2023-11-25T05:04:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.819+0000 7fe5bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.819+0000 7fe5bffff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.819+0000 7fe5bffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.820+0000 7fe5bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.820+0000 7fe5bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.820+0000 7fe5bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.824+0000 7fe5bffff640 10 monclient(hunting): handle_auth_done global_id 4317 payload 274 2023-11-25T05:04:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.824+0000 7fe5bffff640 1 -- 172.21.15.7:0/3596280263 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5c0142bf0 msgr2=0x7fe5c013ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.824+0000 7fe5bffff640 1 --2- 172.21.15.7:0/3596280263 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5c0142bf0 0x7fe5c013ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.824+0000 7fe5bffff640 1 -- 172.21.15.7:0/3596280263 >> v1:172.21.15.90:6789/0 conn(0x7fe5c00a8a30 legacy=0x7fe5c01424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.824+0000 7fe5bffff640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.824+0000 7fe5bffff640 1 monclient: found mon.noname-a 2023-11-25T05:04:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.824+0000 7fe5bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.824+0000 7fe5bffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.824+0000 7fe5bffff640 1 -- 172.21.15.7:0/3596280263 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c00a97f0 con 0x7fe5c00a8e10 2023-11-25T05:04:36.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.828+0000 7fe5bffff640 1 --2- 172.21.15.7:0/3596280263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5c00a8e10 0x7fe5c013f230 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009e50 tx=0x7fe5b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29fe5c674cd01925 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5be7fc640 1 -- 172.21.15.7:0/3596280263 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b802ca70 con 0x7fe5c00a8e10 2023-11-25T05:04:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5be7fc640 10 monclient: dump: 2023-11-25T05:04:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.847 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5be7fc640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5be7fc640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5be7fc640 1 -- 172.21.15.7:0/3596280263 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b802cbd0 con 0x7fe5c00a8e10 2023-11-25T05:04:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5c6015640 5 monclient: authenticate success, global_id 4317 2023-11-25T05:04:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5c6015640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5be7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5c6015640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5be7fc640 1 -- 172.21.15.7:0/3596280263 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b80343f0 con 0x7fe5c00a8e10 2023-11-25T05:04:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5c6015640 4 set_mon_vals no callback set 2023-11-25T05:04:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7fe5be7fc640 10 monclient: dump: 2023-11-25T05:04:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.849 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.849 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.849 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.849 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77af7fe640 1 -- 172.21.15.7:0/4120082866 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b00a6b10 con 0x7f77a80c8a30 2023-11-25T05:04:36.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77af7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:36.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77af7fe640 10 monclient: dump: 2023-11-25T05:04:36.849 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.850 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.850 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.850 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.850 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.850 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.850 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.850 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.850 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.850 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77af7fe640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77af7fe640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77af7fe640 1 -- 172.21.15.7:0/4120082866 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b00a6c70 con 0x7f77a80c8a30 2023-11-25T05:04:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77b5f6c640 5 monclient: authenticate success, global_id 4308 2023-11-25T05:04:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77b5f6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77af7fe640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77b5f6c640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77af7fe640 1 -- 172.21.15.7:0/4120082866 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b00a6ea0 con 0x7f77a80c8a30 2023-11-25T05:04:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77b5f6c640 4 set_mon_vals no callback set 2023-11-25T05:04:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.843+0000 7f77af7fe640 10 monclient: dump: 2023-11-25T05:04:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.852 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.852 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.852 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.852 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.852 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.852 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.852 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe5c6015640 10 monclient: shutdown 2023-11-25T05:04:36.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe5c6015640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe5c6015640 1 -- 172.21.15.7:0/3596280263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5c00a8e10 msgr2=0x7fe5c013f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe5c6015640 1 --2- 172.21.15.7:0/3596280263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5c00a8e10 0x7fe5c013f230 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009e50 tx=0x7fe5b802bed0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe5c6015640 1 -- 172.21.15.7:0/3596280263 shutdown_connections 2023-11-25T05:04:36.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe5c6015640 1 --2- 172.21.15.7:0/3596280263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5c00a8e10 0x7fe5c013f230 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe5c6015640 1 --2- 172.21.15.7:0/3596280263 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5c0142bf0 0x7fe5c013ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe5c6015640 1 -- 172.21.15.7:0/3596280263 >> 172.21.15.7:0/3596280263 conn(0x7fe5c0061a30 msgr2=0x7fe5c00b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7f77b5f6c640 10 monclient: shutdown 2023-11-25T05:04:36.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7f77b5f6c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:36.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7f77b5f6c640 1 -- 172.21.15.7:0/4120082866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a80c8a30 msgr2=0x7f77a80d3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7f77b5f6c640 1 --2- 172.21.15.7:0/4120082866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a80c8a30 0x7f77a80d3610 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f77b0063e60 tx=0x7f77b00a5d00 comp rx=0 tx=0).stop 2023-11-25T05:04:36.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe449ffb640 1 -- 172.21.15.7:0/2066593397 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c00ab70 con 0x7fe454142a20 2023-11-25T05:04:36.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe449ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe449ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:36.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe449ffb640 10 monclient: dump: 2023-11-25T05:04:36.855 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.855 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.855 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.855 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.855 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.855 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.856 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.856 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.856 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.856 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe449ffb640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe449ffb640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe449ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe449ffb640 1 -- 172.21.15.7:0/2066593397 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44c00acd0 con 0x7fe454142a20 2023-11-25T05:04:36.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe45b5e9640 5 monclient: authenticate success, global_id 4314 2023-11-25T05:04:36.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe449ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe449ffb640 1 -- 172.21.15.7:0/2066593397 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c012600 con 0x7fe454142a20 2023-11-25T05:04:36.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe449ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe449ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:36.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7f77b5f6c640 1 -- 172.21.15.7:0/4120082866 shutdown_connections 2023-11-25T05:04:36.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7f77b5f6c640 1 --2- 172.21.15.7:0/4120082866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a80c8a30 0x7f77a80d3610 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7f77b5f6c640 1 -- 172.21.15.7:0/4120082866 >> 172.21.15.7:0/4120082866 conn(0x7f77a80065e0 msgr2=0x7f77a80069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:36.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe449ffb640 10 monclient: dump: 2023-11-25T05:04:36.858 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.858 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.858 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.858 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.858 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.858 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.858 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.858 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.858 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.858 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe44b7fe640 4 set_mon_vals no callback set 2023-11-25T05:04:36.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7f77b5f6c640 1 -- 172.21.15.7:0/4120082866 shutdown_connections 2023-11-25T05:04:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.845+0000 7fe45b5e9640 10 monclient: _renew_subs 2023-11-25T05:04:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.845+0000 7fe45b5e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.845+0000 7fe45b5e9640 1 -- 172.21.15.7:0/2066593397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4541db530 con 0x7fe454142a20 2023-11-25T05:04:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.845+0000 7fe45b5e9640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.845+0000 7fe45b5e9640 10 monclient: _renew_subs 2023-11-25T05:04:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.845+0000 7fe45b5e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.845+0000 7fe45b5e9640 1 -- 172.21.15.7:0/2066593397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4541db080 con 0x7fe454142a20 2023-11-25T05:04:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.845+0000 7fe45b5e9640 1 librados: init done 2023-11-25T05:04:36.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe5c6015640 1 -- 172.21.15.7:0/3596280263 shutdown_connections 2023-11-25T05:04:36.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7fe5c6015640 1 -- 172.21.15.7:0/3596280263 wait complete. 2023-11-25T05:04:36.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.845+0000 7fe5c6015640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.845+0000 7fe5c6015640 10 monclient: monmap: 2023-11-25T05:04:36.861 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.861 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:36.861 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:36.861 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:36.861 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.861 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.861 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:36.861 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:36.861 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:36.862 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.844+0000 7f77b5f6c640 1 -- 172.21.15.7:0/4120082866 wait complete. 2023-11-25T05:04:36.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.845+0000 7f77b5f6c640 10 monclient: build_initial_monmap 2023-11-25T05:04:36.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.845+0000 7f77b5f6c640 10 monclient: monmap: 2023-11-25T05:04:36.862 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:36.862 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:36.862 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:36.862 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:36.862 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:36.862 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.863 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:36.863 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:36.863 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:36.863 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.845+0000 7fe5c6015640 1 librados: starting msgr at 2023-11-25T05:04:36.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.846+0000 7fe5c6015640 1 librados: starting objecter 2023-11-25T05:04:36.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.846+0000 7fe5c6015640 1 Processor -- start 2023-11-25T05:04:36.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.846+0000 7f77b5f6c640 1 librados: starting msgr at 2023-11-25T05:04:36.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.846+0000 7f77b5f6c640 1 librados: starting objecter 2023-11-25T05:04:36.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.846+0000 7f77b5f6c640 1 Processor -- start 2023-11-25T05:04:36.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.846+0000 7fe5c6015640 1 -- start start 2023-11-25T05:04:36.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.846+0000 7fe5c6015640 1 librados: setting wanted keys 2023-11-25T05:04:36.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.846+0000 7fe5c6015640 1 librados: calling monclient init 2023-11-25T05:04:36.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.846+0000 7fe5c6015640 10 monclient: init 2023-11-25T05:04:36.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.846+0000 7f77b5f6c640 1 -- start start 2023-11-25T05:04:36.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.846+0000 7f77b5f6c640 1 librados: setting wanted keys 2023-11-25T05:04:36.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.846+0000 7f77b5f6c640 1 librados: calling monclient init 2023-11-25T05:04:36.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.846+0000 7f77b5f6c640 10 monclient: init 2023-11-25T05:04:36.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7fe45b5e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7fe45b5e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7fe45b5e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7fe45b5e9640 1 -- 172.21.15.7:0/2066593397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe454067ac0 con 0x7fe454142a20 2023-11-25T05:04:36.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:04:36.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f77a8038d80 0x7f77a802fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 10 monclient(hunting): picked mon.noname-c con 0x7f77a8038d80 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a80c8a30 0x7f77a8160940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 10 monclient(hunting): picked mon.noname-b con 0x7f77a80c8a30 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:36.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a80c2c30 0x7f77a80c30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 10 monclient(hunting): picked mon.noname-a con 0x7f77a80c2c30 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:36.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a8161280 con 0x7f77a80c2c30 2023-11-25T05:04:36.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a80c35f0 con 0x7f77a80c8a30 2023-11-25T05:04:36.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a80c37b0 con 0x7f77a8038d80 2023-11-25T05:04:36.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7f77b5f6c640 10 monclient(hunting): authenticate will time out at 1403.090775s 2023-11-25T05:04:36.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.847+0000 7fe5c6015640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:36.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.848+0000 7fe5c6015640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-25T05:04:36.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.848+0000 7fe5c6015640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5c00a8a30 0x7fe5c006da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.848+0000 7fe5c6015640 10 monclient(hunting): picked mon.noname-c con 0x7fe5c00a8a30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:36.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.848+0000 7fe5c6015640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5c00a8e10 0x7fe5c01e74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.848+0000 7fe5c6015640 10 monclient(hunting): picked mon.noname-a con 0x7fe5c00a8e10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:36.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.848+0000 7fe5c6015640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5c0142bf0 0x7fe5c01e79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.848+0000 7fe5c6015640 10 monclient(hunting): picked mon.noname-b con 0x7fe5c0142bf0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:36.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.848+0000 7fe5c6015640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.848+0000 7fe5c6015640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c0144ba0 con 0x7fe5c00a8e10 2023-11-25T05:04:36.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.848+0000 7fe5c6015640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.848+0000 7fe5c6015640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c01e6f90 con 0x7fe5c0142bf0 2023-11-25T05:04:36.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.848+0000 7fe5c6015640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:36.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.849+0000 7fe5c6015640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c0143130 con 0x7fe5c00a8a30 2023-11-25T05:04:36.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.849+0000 7fe5c6015640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:36.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.849+0000 7fe5c6015640 10 monclient(hunting): authenticate will time out at 1403.091976s 2023-11-25T05:04:36.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.851+0000 7ff72effd640 1 -- 172.21.15.7:0/4102865996 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff73401b460 con 0x7ff7400b29e0 2023-11-25T05:04:36.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.852+0000 7ff72effd640 1 --2- 172.21.15.7:0/4102865996 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ff72006c820 0x7ff72006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.852+0000 7ff73e59c640 1 --2- 172.21.15.7:0/4102865996 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ff72006c820 0x7ff72006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.852+0000 7ff73e59c640 10 monclient: get_auth_request con 0x7ff72006c820 auth_method 0 2023-11-25T05:04:36.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.853+0000 7ff72effd640 10 client.4286.objecter ms_handle_connect 0x7ff72006c820 2023-11-25T05:04:36.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.853+0000 7ff73e59c640 1 --2- 172.21.15.7:0/4102865996 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ff72006c820 0x7ff72006ece0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff730000f60 tx=0x7ff7300079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.878+0000 7fd3a6a0e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-11-25T05:04:36.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.878+0000 7fd3a6a0e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-11-25T05:04:36.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.878+0000 7fd3a6a0e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:36.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.878+0000 7fd3a6a0e640 1 -- 172.21.15.7:0/758239465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd3a00adec0 con 0x7fd3a00a8a30 2023-11-25T05:04:36.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.878+0000 7ff72effd640 1 -- 172.21.15.7:0/4102865996 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff734059430 con 0x7ff7400b29e0 2023-11-25T05:04:36.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.878+0000 7ff72effd640 10 client.4286.objecter ms_dispatch 0x7ff740000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:36.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.878+0000 7ff72effd640 3 client.4286.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:36.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.878+0000 7ff72effd640 3 client.4286.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:36.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.879+0000 7ff72effd640 20 client.4286.objecter dump_active .. 0 homeless 2023-11-25T05:04:36.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.879+0000 7ff72effd640 1 -- 172.21.15.7:0/4102865996 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff734055070 con 0x7ff7400b29e0 2023-11-25T05:04:36.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.879+0000 7ff72effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.879+0000 7ff72effd640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:36.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.887+0000 7fe5bf7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5c00a8a30 0x7fe5c006da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f917fa640 1 -- 172.21.15.7:0/3084147837 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8403ac60 con 0x7f4f940a8a30 2023-11-25T05:04:36.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f917fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:36.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f917fa640 10 monclient: dump: 2023-11-25T05:04:36.890 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.890 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.891 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.891 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.891 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.891 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.891 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.891 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.891 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.891 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f917fa640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f917fa640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f917fa640 1 -- 172.21.15.7:0/3084147837 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8403adc0 con 0x7f4f940a8a30 2023-11-25T05:04:36.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f9add5640 5 monclient: authenticate success, global_id 4289 2023-11-25T05:04:36.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f9add5640 10 monclient: _renew_subs 2023-11-25T05:04:36.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f9add5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f9add5640 1 -- 172.21.15.7:0/3084147837 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f940b15d0 con 0x7f4f940a8a30 2023-11-25T05:04:36.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f917fa640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f9add5640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f9add5640 10 monclient: _renew_subs 2023-11-25T05:04:36.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f917fa640 1 -- 172.21.15.7:0/3084147837 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84044630 con 0x7f4f940a8a30 2023-11-25T05:04:36.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f9add5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f92ffd640 4 set_mon_vals no callback set 2023-11-25T05:04:36.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f9add5640 1 -- 172.21.15.7:0/3084147837 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9413c050 con 0x7f4f940a8a30 2023-11-25T05:04:36.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f9add5640 1 librados: init done 2023-11-25T05:04:36.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:36.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.888+0000 7f4f917fa640 10 monclient: dump: 2023-11-25T05:04:36.893 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.893 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.893 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.893 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.893 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.893 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.893 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.894 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.894 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.894 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.889+0000 7f4f9add5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.889+0000 7f4f9add5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.889+0000 7f4f9add5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.889+0000 7f77b4f6a640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f77a8038d80 0x7f77a802fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.889+0000 7f4f9add5640 1 -- 172.21.15.7:0/3084147837 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f94067ac0 con 0x7f4f940a8a30 2023-11-25T05:04:36.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.890+0000 7fe5bf7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5c00a8a30 0x7fe5c006da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41790/0 (socket says 172.21.15.7:41790) 2023-11-25T05:04:36.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.890+0000 7fe5bf7fe640 1 -- 172.21.15.7:0/1156455543 learned_addr learned my addr 172.21.15.7:0/1156455543 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.890+0000 7fe5bf7fe640 10 monclient(hunting): get_auth_request con 0x7fe5c00a8a30 auth_method 0 2023-11-25T05:04:36.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.890+0000 7fe5bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.890+0000 7fe5bf7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.890+0000 7fe5bf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.891+0000 7fe5bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.891+0000 7fe5bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.891+0000 7fe5bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.891+0000 7f77b4f6a640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f77a8038d80 0x7f77a802fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41780/0 (socket says 172.21.15.7:41780) 2023-11-25T05:04:36.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.891+0000 7f77b4f6a640 1 -- 172.21.15.7:0/214933219 learned_addr learned my addr 172.21.15.7:0/214933219 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:36.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.891+0000 7f77b4f6a640 10 monclient(hunting): get_auth_request con 0x7f77a8038d80 auth_method 0 2023-11-25T05:04:36.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.891+0000 7f77b4f6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:36.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.891+0000 7f77b4f6a640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:36.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.891+0000 7f77b4f6a640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:36.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.894+0000 7fe5bf7fe640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2023-11-25T05:04:36.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.894+0000 7fe5bf7fe640 1 -- 172.21.15.7:0/1156455543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5c0142bf0 msgr2=0x7fe5c01e79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.894+0000 7fe5bf7fe640 1 --2- 172.21.15.7:0/1156455543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5c0142bf0 0x7fe5c01e79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.894+0000 7fe5bf7fe640 1 -- 172.21.15.7:0/1156455543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5c00a8e10 msgr2=0x7fe5c01e74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.894+0000 7fe5bf7fe640 1 --2- 172.21.15.7:0/1156455543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5c00a8e10 0x7fe5c01e74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.894+0000 7fe5bf7fe640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.894+0000 7fe5bf7fe640 1 monclient: found mon.noname-c 2023-11-25T05:04:36.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.894+0000 7fe5bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.895+0000 7fe5bf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.895+0000 7fe5bf7fe640 1 -- 172.21.15.7:0/1156455543 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b8009b30 con 0x7fe5c00a8a30 2023-11-25T05:04:36.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.896+0000 7f77b4f6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:36.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.897+0000 7f77b4f6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:36.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.897+0000 7f77b4f6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:36.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.902+0000 7f77b4f6a640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2023-11-25T05:04:36.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.902+0000 7f77b4f6a640 1 -- 172.21.15.7:0/214933219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a80c8a30 msgr2=0x7f77a8160940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.902+0000 7f77b4f6a640 1 --2- 172.21.15.7:0/214933219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a80c8a30 0x7f77a8160940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.902+0000 7f77b4f6a640 1 -- 172.21.15.7:0/214933219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a80c2c30 msgr2=0x7f77a80c30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:36.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.902+0000 7f77b4f6a640 1 --2- 172.21.15.7:0/214933219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a80c2c30 0x7f77a80c30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:36.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.902+0000 7f77b4f6a640 10 monclient: _finish_hunting 0 2023-11-25T05:04:36.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.902+0000 7f77b4f6a640 1 monclient: found mon.noname-c 2023-11-25T05:04:36.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.902+0000 7f77b4f6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:36.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.902+0000 7f77b4f6a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.902+0000 7f77b4f6a640 1 -- 172.21.15.7:0/214933219 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a8160e80 con 0x7f77a8038d80 2023-11-25T05:04:36.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.904+0000 7fe5bf7fe640 1 --2- 172.21.15.7:0/1156455543 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5c00a8a30 0x7fe5c006da90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe5b000c410 tx=0x7fe5b000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.904+0000 7fe5a7fff640 10 client.?.objecter ms_handle_connect 0x7fe5c00a8a30 2023-11-25T05:04:36.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.904+0000 7fe5a7fff640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:36.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.907+0000 7f77b4f6a640 1 --2- 172.21.15.7:0/214933219 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f77a8038d80 0x7f77a802fc00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f77a400c4a0 tx=0x7f77a400c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.907+0000 7f77ad7fa640 10 client.?.objecter ms_handle_connect 0x7f77a8038d80 2023-11-25T05:04:36.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.907+0000 7f77ad7fa640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:36.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5a7fff640 1 -- 172.21.15.7:0/1156455543 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b00035f0 con 0x7fe5c00a8a30 2023-11-25T05:04:36.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5a7fff640 10 monclient: dump: 2023-11-25T05:04:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.917 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5a7fff640 10 monclient: _finish_auth 0 2023-11-25T05:04:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5a7fff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5a7fff640 1 -- 172.21.15.7:0/1156455543 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b0003750 con 0x7fe5c00a8a30 2023-11-25T05:04:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5c6015640 5 monclient: authenticate success, global_id 4292 2023-11-25T05:04:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5c6015640 10 monclient: _renew_subs 2023-11-25T05:04:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5c6015640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5c6015640 1 -- 172.21.15.7:0/1156455543 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c006dfd0 con 0x7fe5c00a8a30 2023-11-25T05:04:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5a7fff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5a7fff640 1 -- 172.21.15.7:0/1156455543 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0003b80 con 0x7fe5c00a8a30 2023-11-25T05:04:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5a7fff640 10 monclient: dump: 2023-11-25T05:04:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:36.919 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.912+0000 7fe5bdffb640 4 set_mon_vals no callback set 2023-11-25T05:04:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.913+0000 7fe5c6015640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.913+0000 7fe5c6015640 10 monclient: _renew_subs 2023-11-25T05:04:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.913+0000 7fe5c6015640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.913+0000 7fe5c6015640 1 -- 172.21.15.7:0/1156455543 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c0145950 con 0x7fe5c00a8a30 2023-11-25T05:04:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.913+0000 7fe5c6015640 1 librados: init done 2023-11-25T05:04:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.914+0000 7fe5c6015640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.914+0000 7fe5c6015640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.914+0000 7fe5c6015640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.914+0000 7fe5c6015640 1 -- 172.21.15.7:0/1156455543 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c006e1c0 con 0x7fe5c00a8a30 2023-11-25T05:04:36.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.933+0000 7f26a1d01640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-11-25T05:04:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.933+0000 7f26a1d01640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-11-25T05:04:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.933+0000 7f26a1d01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.933+0000 7f26a1d01640 1 -- 172.21.15.7:0/3597088794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f269c0ac4a0 con 0x7f269c0a8a30 2023-11-25T05:04:36.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.945+0000 7f523cff9640 1 -- 172.21.15.7:0/790342916 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5234059150 con 0x7f52400a8e10 2023-11-25T05:04:36.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.945+0000 7f523cff9640 10 client.4345.objecter ms_dispatch 0x7f5240000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:36.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.945+0000 7f523cff9640 3 client.4345.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:36.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.945+0000 7f523cff9640 3 client.4345.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:36.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.945+0000 7f523cff9640 20 client.4345.objecter dump_active .. 0 homeless 2023-11-25T05:04:36.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.957+0000 7fe5a7fff640 1 -- 172.21.15.7:0/1156455543 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe5b0003ce0 con 0x7fe5c00a8a30 2023-11-25T05:04:36.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.957+0000 7f523cff9640 1 -- 172.21.15.7:0/790342916 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5234054d90 con 0x7f52400a8e10 2023-11-25T05:04:36.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.957+0000 7f523cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:36.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.957+0000 7f523cff9640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:36.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.957+0000 7fe5a7fff640 1 --2- 172.21.15.7:0/1156455543 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe5ac06c850 0x7fe5ac06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:36.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.957+0000 7fe5beffd640 1 --2- 172.21.15.7:0/1156455543 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe5ac06c850 0x7fe5ac06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:36.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.958+0000 7fe5beffd640 10 monclient: get_auth_request con 0x7fe5ac06c850 auth_method 0 2023-11-25T05:04:36.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.958+0000 7fe5a7fff640 10 client.4292.objecter ms_handle_connect 0x7fe5ac06c850 2023-11-25T05:04:36.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.958+0000 7fe5beffd640 1 --2- 172.21.15.7:0/1156455543 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe5ac06c850 0x7fe5ac06ed10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe5c013c830 tx=0x7fe5b4006bf0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:36.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.962+0000 7fe5a7fff640 1 -- 172.21.15.7:0/1156455543 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe5b0058500 con 0x7fe5c00a8a30 2023-11-25T05:04:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.962+0000 7fe5a7fff640 10 client.4292.objecter ms_dispatch 0x7fe5c0000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.962+0000 7fe5a7fff640 3 client.4292.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.962+0000 7fe5a7fff640 3 client.4292.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:36.962+0000 7fe5a7fff640 20 client.4292.objecter dump_active .. 0 homeless 2023-11-25T05:04:37.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.026+0000 7fd39cff9640 1 -- 172.21.15.7:0/758239465 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd390057720 con 0x7fd3a00a8a30 2023-11-25T05:04:37.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.026+0000 7fd39cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-11-25T05:04:37.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.026+0000 7fd39cff9640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:37.030 INFO:teuthology.orchestra.run.smithi007.stdout:55834574851 2023-11-25T05:04:37.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.028+0000 7fd3a6a0e640 10 librados: watch_flush enter 2023-11-25T05:04:37.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.028+0000 7fd3a6a0e640 10 librados: watch_flush exit 2023-11-25T05:04:37.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.028+0000 7fd3a6a0e640 20 client.4339.objecter shutdown clearing up homeless session... 2023-11-25T05:04:37.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.028+0000 7fd3a6a0e640 10 client.4339.objecter successfully canceled tick 2023-11-25T05:04:37.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.029+0000 7fd3a6a0e640 1 -- 172.21.15.7:0/758239465 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fd38c06c820 msgr2=0x7fd38c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:37.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.029+0000 7fd3a6a0e640 1 --2- 172.21.15.7:0/758239465 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fd38c06c820 0x7fd38c06ece0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0072c60 tx=0x7fd394006bf0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.029+0000 7fd3a6a0e640 10 monclient: shutdown 2023-11-25T05:04:37.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.029+0000 7fd3a6a0e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:37.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.029+0000 7fd3a6a0e640 1 -- 172.21.15.7:0/758239465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a00a8a30 msgr2=0x7fd3a0145500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:37.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.029+0000 7fd3a6a0e640 1 --2- 172.21.15.7:0/758239465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a00a8a30 0x7fd3a0145500 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd39000c4a0 tx=0x7fd39000c960 comp rx=0 tx=0).stop 2023-11-25T05:04:37.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.029+0000 7fd3a6a0e640 1 -- 172.21.15.7:0/758239465 shutdown_connections 2023-11-25T05:04:37.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.029+0000 7fd3a6a0e640 1 --2- 172.21.15.7:0/758239465 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fd38c06c820 0x7fd38c06ece0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.029+0000 7fd3a6a0e640 1 --2- 172.21.15.7:0/758239465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a0142bf0 0x7fd3a00ad680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.029+0000 7fd3a6a0e640 1 --2- 172.21.15.7:0/758239465 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3a00a8e10 0x7fd3a00ad060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.029+0000 7fd3a6a0e640 1 --2- 172.21.15.7:0/758239465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a00a8a30 0x7fd3a0145500 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.029+0000 7fd3a6a0e640 1 -- 172.21.15.7:0/758239465 >> 172.21.15.7:0/758239465 conn(0x7fd3a0061a30 msgr2=0x7fd3a01424e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:37.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.029+0000 7fd3a6a0e640 1 -- 172.21.15.7:0/758239465 shutdown_connections 2023-11-25T05:04:37.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.029+0000 7fd3a6a0e640 1 -- 172.21.15.7:0/758239465 wait complete. 2023-11-25T05:04:37.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.029+0000 7fd3a6a0e640 1 librados: shutdown 2023-11-25T05:04:37.047 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.5 2023-11-25T05:04:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.094+0000 7ff74487c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-11-25T05:04:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.094+0000 7ff74487c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-11-25T05:04:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.094+0000 7ff74487c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.094+0000 7ff74487c640 1 -- 172.21.15.7:0/4102865996 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff740075b60 con 0x7ff7400b29e0 2023-11-25T05:04:37.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.137+0000 7f5246a1f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-11-25T05:04:37.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.137+0000 7f5246a1f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-11-25T05:04:37.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.137+0000 7f5246a1f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:37.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.137+0000 7f5246a1f640 1 -- 172.21.15.7:0/790342916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5240075b60 con 0x7f52400a8e10 2023-11-25T05:04:37.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.140+0000 7f523cff9640 1 -- 172.21.15.7:0/790342916 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5234015020 con 0x7f52400a8e10 2023-11-25T05:04:37.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.140+0000 7f523cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-11-25T05:04:37.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.140+0000 7f523cff9640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:37.142 INFO:teuthology.orchestra.run.smithi007.stdout:68719476740 2023-11-25T05:04:37.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 10 librados: watch_flush enter 2023-11-25T05:04:37.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 10 librados: watch_flush exit 2023-11-25T05:04:37.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 20 client.4345.objecter shutdown clearing up homeless session... 2023-11-25T05:04:37.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 10 client.4345.objecter successfully canceled tick 2023-11-25T05:04:37.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 1 -- 172.21.15.7:0/790342916 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f522c06c820 msgr2=0x7f522c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:37.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 1 --2- 172.21.15.7:0/790342916 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f522c06c820 0x7f522c06ece0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5230001560 tx=0x7f5230007990 comp rx=0 tx=0).stop 2023-11-25T05:04:37.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 10 monclient: shutdown 2023-11-25T05:04:37.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:37.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 1 -- 172.21.15.7:0/790342916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f52400a8e10 msgr2=0x7f52400ad9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:37.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 1 --2- 172.21.15.7:0/790342916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f52400a8e10 0x7f52400ad9c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f523400b410 tx=0x7f523400b8d0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 1 -- 172.21.15.7:0/790342916 shutdown_connections 2023-11-25T05:04:37.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 1 --2- 172.21.15.7:0/790342916 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f522c06c820 0x7f522c06ece0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 1 --2- 172.21.15.7:0/790342916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5240142bf0 0x7f52401da960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 1 --2- 172.21.15.7:0/790342916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f52400a8e10 0x7f52400ad9c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 1 --2- 172.21.15.7:0/790342916 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52400a8a30 0x7f52400b0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 1 -- 172.21.15.7:0/790342916 >> 172.21.15.7:0/790342916 conn(0x7f5240061a30 msgr2=0x7f524013cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:37.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 1 -- 172.21.15.7:0/790342916 shutdown_connections 2023-11-25T05:04:37.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 1 -- 172.21.15.7:0/790342916 wait complete. 2023-11-25T05:04:37.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7f5246a1f640 1 librados: shutdown 2023-11-25T05:04:37.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7fe449ffb640 1 -- 172.21.15.7:0/2066593397 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe44c007070 con 0x7fe454142a20 2023-11-25T05:04:37.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.143+0000 7fe449ffb640 1 --2- 172.21.15.7:0/2066593397 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe43806c850 0x7fe43806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:37.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.144+0000 7fe459385640 1 --2- 172.21.15.7:0/2066593397 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe43806c850 0x7fe43806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:37.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.144+0000 7fe459385640 10 monclient: get_auth_request con 0x7fe43806c850 auth_method 0 2023-11-25T05:04:37.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.144+0000 7fe449ffb640 10 client.4314.objecter ms_handle_connect 0x7fe43806c850 2023-11-25T05:04:37.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.144+0000 7fe459385640 1 --2- 172.21.15.7:0/2066593397 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe43806c850 0x7fe43806ed10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe450036000 tx=0x7fe45003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:37.160 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476740 for osd.1 2023-11-25T05:04:37.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.164+0000 7fe449ffb640 1 -- 172.21.15.7:0/2066593397 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe44c0586d0 con 0x7fe454142a20 2023-11-25T05:04:37.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.164+0000 7fe449ffb640 10 client.4314.objecter ms_dispatch 0x7fe454000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:37.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.164+0000 7fe449ffb640 3 client.4314.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:37.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.164+0000 7fe449ffb640 3 client.4314.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:37.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.165+0000 7fe449ffb640 20 client.4314.objecter dump_active .. 0 homeless 2023-11-25T05:04:37.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.165+0000 7fe449ffb640 1 -- 172.21.15.7:0/2066593397 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe44c054a10 con 0x7fe454142a20 2023-11-25T05:04:37.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.165+0000 7fe449ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:37.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.165+0000 7fe449ffb640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:37.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.186+0000 7f2683fff640 1 -- 172.21.15.7:0/3597088794 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f269c0ac4a0 con 0x7f269c0a8a30 2023-11-25T05:04:37.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.186+0000 7f2683fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-11-25T05:04:37.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.186+0000 7f2683fff640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:37.188 INFO:teuthology.orchestra.run.smithi007.stdout:68719476739 2023-11-25T05:04:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 10 librados: watch_flush enter 2023-11-25T05:04:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 10 librados: watch_flush exit 2023-11-25T05:04:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 20 client.4305.objecter shutdown clearing up homeless session... 2023-11-25T05:04:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 10 client.4305.objecter successfully canceled tick 2023-11-25T05:04:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 1 -- 172.21.15.7:0/3597088794 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f268806c850 msgr2=0x7f268806ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 1 --2- 172.21.15.7:0/3597088794 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f268806c850 0x7f268806ed10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f268c036000 tx=0x7f268c008030 comp rx=0 tx=0).stop 2023-11-25T05:04:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 10 monclient: shutdown 2023-11-25T05:04:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 1 -- 172.21.15.7:0/3597088794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f269c0a8a30 msgr2=0x7f269c0b6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 1 --2- 172.21.15.7:0/3597088794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f269c0a8a30 0x7f269c0b6e40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f269000c390 tx=0x7f269000c850 comp rx=0 tx=0).stop 2023-11-25T05:04:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 1 -- 172.21.15.7:0/3597088794 shutdown_connections 2023-11-25T05:04:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 1 --2- 172.21.15.7:0/3597088794 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f268806c850 0x7f268806ed10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 1 --2- 172.21.15.7:0/3597088794 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f269c142bf0 0x7f269c0b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 1 --2- 172.21.15.7:0/3597088794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f269c0a8e10 0x7f269c145980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 1 --2- 172.21.15.7:0/3597088794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f269c0a8a30 0x7f269c0b6e40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 1 -- 172.21.15.7:0/3597088794 >> 172.21.15.7:0/3597088794 conn(0x7f269c061a30 msgr2=0x7f269c0b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 1 -- 172.21.15.7:0/3597088794 shutdown_connections 2023-11-25T05:04:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 1 -- 172.21.15.7:0/3597088794 wait complete. 2023-11-25T05:04:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.188+0000 7f26a1d01640 1 librados: shutdown 2023-11-25T05:04:37.201 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476739 for osd.3 2023-11-25T05:04:37.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.236+0000 7fe5a7fff640 1 -- 172.21.15.7:0/1156455543 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5b0054140 con 0x7fe5c00a8a30 2023-11-25T05:04:37.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.236+0000 7fe5a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:37.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.236+0000 7fe5a7fff640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:37.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77ad7fa640 1 -- 172.21.15.7:0/214933219 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a4005ce0 con 0x7f77a8038d80 2023-11-25T05:04:37.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:37.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77ad7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:37.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77ad7fa640 10 monclient: dump: 2023-11-25T05:04:37.250 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:37.250 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:37.250 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:37.250 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:37.250 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:37.250 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:37.250 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:37.251 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:37.251 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:37.251 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:37.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77ad7fa640 10 monclient: _finish_auth 0 2023-11-25T05:04:37.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77ad7fa640 10 monclient: _check_auth_tickets 2023-11-25T05:04:37.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:37.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77ad7fa640 1 -- 172.21.15.7:0/214933219 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a4005e40 con 0x7f77a8038d80 2023-11-25T05:04:37.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77ad7fa640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:37.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77ad7fa640 1 -- 172.21.15.7:0/214933219 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a4007270 con 0x7f77a8038d80 2023-11-25T05:04:37.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77b5f6c640 5 monclient: authenticate success, global_id 4295 2023-11-25T05:04:37.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77b5f6c640 10 monclient: _renew_subs 2023-11-25T05:04:37.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77b5f6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:37.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77b5f6c640 1 -- 172.21.15.7:0/214933219 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a80a61f0 con 0x7f77a8038d80 2023-11-25T05:04:37.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77aeffd640 4 set_mon_vals no callback set 2023-11-25T05:04:37.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:37.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77ad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:37.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77ad7fa640 10 monclient: dump: 2023-11-25T05:04:37.252 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:37.252 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:37.252 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:37.252 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:37.252 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:37.252 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:37.252 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:37.253 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:37.253 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:37.253 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:37.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77b5f6c640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:37.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77b5f6c640 10 monclient: _renew_subs 2023-11-25T05:04:37.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77b5f6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:37.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77b5f6c640 1 -- 172.21.15.7:0/214933219 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a815d300 con 0x7f77a8038d80 2023-11-25T05:04:37.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.248+0000 7f77b5f6c640 1 librados: init done 2023-11-25T05:04:37.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.249+0000 7f77b5f6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:37.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.249+0000 7f77b5f6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:37.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.249+0000 7f77b5f6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:37.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.249+0000 7f77b5f6c640 1 -- 172.21.15.7:0/214933219 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a81610d0 con 0x7f77a8038d80 2023-11-25T05:04:37.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.351+0000 7fe45b5e9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-11-25T05:04:37.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.351+0000 7fe45b5e9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-11-25T05:04:37.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.351+0000 7fe45b5e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:37.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.351+0000 7fe45b5e9640 1 -- 172.21.15.7:0/2066593397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fe4541db330 con 0x7fe454142a20 2023-11-25T05:04:37.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.357+0000 7fe449ffb640 1 -- 172.21.15.7:0/2066593397 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe44c017070 con 0x7fe454142a20 2023-11-25T05:04:37.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.357+0000 7fe449ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-11-25T05:04:37.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.357+0000 7fe449ffb640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:37.358 INFO:teuthology.orchestra.run.smithi007.stdout:64424509445 2023-11-25T05:04:37.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 10 librados: watch_flush enter 2023-11-25T05:04:37.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 10 librados: watch_flush exit 2023-11-25T05:04:37.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 20 client.4314.objecter shutdown clearing up homeless session... 2023-11-25T05:04:37.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 10 client.4314.objecter successfully canceled tick 2023-11-25T05:04:37.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 1 -- 172.21.15.7:0/2066593397 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe43806c850 msgr2=0x7fe43806ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:37.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 1 --2- 172.21.15.7:0/2066593397 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe43806c850 0x7fe43806ed10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe450036000 tx=0x7fe45003b040 comp rx=0 tx=0).stop 2023-11-25T05:04:37.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 10 monclient: shutdown 2023-11-25T05:04:37.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:37.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 1 -- 172.21.15.7:0/2066593397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe454142a20 msgr2=0x7fe45413b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:37.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 1 --2- 172.21.15.7:0/2066593397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe454142a20 0x7fe45413b1f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00c9f0 tx=0x7fe44c00ceb0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 1 -- 172.21.15.7:0/2066593397 shutdown_connections 2023-11-25T05:04:37.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 1 --2- 172.21.15.7:0/2066593397 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe43806c850 0x7fe43806ed10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 1 --2- 172.21.15.7:0/2066593397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe454142a20 0x7fe45413b1f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 1 --2- 172.21.15.7:0/2066593397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4540a8e10 0x7fe45413acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 1 --2- 172.21.15.7:0/2066593397 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe4540a8a30 0x7fe4541dab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 1 -- 172.21.15.7:0/2066593397 >> 172.21.15.7:0/2066593397 conn(0x7fe454061a30 msgr2=0x7fe4540b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:37.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 1 -- 172.21.15.7:0/2066593397 shutdown_connections 2023-11-25T05:04:37.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 1 -- 172.21.15.7:0/2066593397 wait complete. 2023-11-25T05:04:37.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.359+0000 7fe45b5e9640 1 librados: shutdown 2023-11-25T05:04:37.373 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.4 2023-11-25T05:04:37.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.461+0000 7fe5c6015640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-11-25T05:04:37.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.461+0000 7fe5c6015640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-11-25T05:04:37.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.461+0000 7fe5c6015640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:37.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.461+0000 7fe5c6015640 1 -- 172.21.15.7:0/1156455543 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe5c0067ac0 con 0x7fe5c00a8a30 2023-11-25T05:04:37.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.881+0000 7ff72effd640 1 -- 172.21.15.7:0/4102865996 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff734058690 con 0x7ff7400b29e0 2023-11-25T05:04:37.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.881+0000 7ff72effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-11-25T05:04:37.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.881+0000 7ff72effd640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:37.883 INFO:teuthology.orchestra.run.smithi007.stdout:68719476739 2023-11-25T05:04:37.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.882+0000 7ff74487c640 10 librados: watch_flush enter 2023-11-25T05:04:37.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.882+0000 7ff74487c640 10 librados: watch_flush exit 2023-11-25T05:04:37.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 20 client.4286.objecter shutdown clearing up homeless session... 2023-11-25T05:04:37.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 10 client.4286.objecter successfully canceled tick 2023-11-25T05:04:37.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 1 -- 172.21.15.7:0/4102865996 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ff72006c820 msgr2=0x7ff72006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:37.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 1 --2- 172.21.15.7:0/4102865996 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ff72006c820 0x7ff72006ece0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff730000f60 tx=0x7ff7300079b0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 10 monclient: shutdown 2023-11-25T05:04:37.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:37.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 1 -- 172.21.15.7:0/4102865996 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff7400b29e0 msgr2=0x7ff740133ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:37.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 1 --2- 172.21.15.7:0/4102865996 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff7400b29e0 0x7ff740133ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff73400b4a0 tx=0x7ff73400b960 comp rx=0 tx=0).stop 2023-11-25T05:04:37.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 1 -- 172.21.15.7:0/4102865996 shutdown_connections 2023-11-25T05:04:37.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 1 --2- 172.21.15.7:0/4102865996 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ff72006c820 0x7ff72006ece0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 1 --2- 172.21.15.7:0/4102865996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff740143780 0x7ff7400aca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 1 --2- 172.21.15.7:0/4102865996 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff7400b29e0 0x7ff740133ec0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 1 --2- 172.21.15.7:0/4102865996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7400b0f20 0x7ff7400abec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 1 -- 172.21.15.7:0/4102865996 >> 172.21.15.7:0/4102865996 conn(0x7ff740061a30 msgr2=0x7ff7400b2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:37.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 1 -- 172.21.15.7:0/4102865996 shutdown_connections 2023-11-25T05:04:37.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 1 -- 172.21.15.7:0/4102865996 wait complete. 2023-11-25T05:04:37.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.883+0000 7ff74487c640 1 librados: shutdown 2023-11-25T05:04:37.895 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476739 for osd.0 2023-11-25T05:04:37.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.921+0000 7f4f917fa640 1 -- 172.21.15.7:0/3084147837 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4f8402f050 con 0x7f4f940a8a30 2023-11-25T05:04:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.921+0000 7f4f917fa640 1 --2- 172.21.15.7:0/3084147837 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f4f8006c800 0x7f4f8006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.922+0000 7f4f93fff640 1 --2- 172.21.15.7:0/3084147837 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f4f8006c800 0x7f4f8006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.922+0000 7f4f93fff640 10 monclient: get_auth_request con 0x7f4f8006c800 auth_method 0 2023-11-25T05:04:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.922+0000 7f4f917fa640 10 client.4289.objecter ms_handle_connect 0x7f4f8006c800 2023-11-25T05:04:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.922+0000 7f4f93fff640 1 --2- 172.21.15.7:0/3084147837 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f4f8006c800 0x7f4f8006ecc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4f941d4c70 tx=0x7f4f88007d10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:37.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.935+0000 7f4f917fa640 1 -- 172.21.15.7:0/3084147837 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4f8407ae10 con 0x7f4f940a8a30 2023-11-25T05:04:37.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.935+0000 7f4f917fa640 10 client.4289.objecter ms_dispatch 0x7f4f94000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:37.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.935+0000 7f4f917fa640 3 client.4289.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:37.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.935+0000 7f4f917fa640 3 client.4289.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:37.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.935+0000 7f4f917fa640 20 client.4289.objecter dump_active .. 0 homeless 2023-11-25T05:04:37.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.935+0000 7f4f917fa640 1 -- 172.21.15.7:0/3084147837 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f84076a50 con 0x7f4f940a8a30 2023-11-25T05:04:37.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.935+0000 7f4f917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:37.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.935+0000 7f4f917fa640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:37.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.935+0000 7fe5a7fff640 1 -- 172.21.15.7:0/1156455543 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe5b0017020 con 0x7fe5c00a8a30 2023-11-25T05:04:37.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.935+0000 7fe5a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-11-25T05:04:37.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.935+0000 7fe5a7fff640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:37.939 INFO:teuthology.orchestra.run.smithi007.stdout:64424509444 2023-11-25T05:04:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 10 librados: watch_flush enter 2023-11-25T05:04:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 10 librados: watch_flush exit 2023-11-25T05:04:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 20 client.4292.objecter shutdown clearing up homeless session... 2023-11-25T05:04:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 10 client.4292.objecter successfully canceled tick 2023-11-25T05:04:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 1 -- 172.21.15.7:0/1156455543 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe5ac06c850 msgr2=0x7fe5ac06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 1 --2- 172.21.15.7:0/1156455543 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe5ac06c850 0x7fe5ac06ed10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe5c013c830 tx=0x7fe5b4006bf0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 10 monclient: shutdown 2023-11-25T05:04:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 1 -- 172.21.15.7:0/1156455543 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5c00a8a30 msgr2=0x7fe5c006da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 1 --2- 172.21.15.7:0/1156455543 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5c00a8a30 0x7fe5c006da90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe5b000c410 tx=0x7fe5b000c8d0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 1 -- 172.21.15.7:0/1156455543 shutdown_connections 2023-11-25T05:04:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 1 --2- 172.21.15.7:0/1156455543 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fe5ac06c850 0x7fe5ac06ed10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 1 --2- 172.21.15.7:0/1156455543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5c0142bf0 0x7fe5c01e79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 1 --2- 172.21.15.7:0/1156455543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5c00a8e10 0x7fe5c01e74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 1 --2- 172.21.15.7:0/1156455543 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5c00a8a30 0x7fe5c006da90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 1 -- 172.21.15.7:0/1156455543 >> 172.21.15.7:0/1156455543 conn(0x7fe5c0061a30 msgr2=0x7fe5c013dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 1 -- 172.21.15.7:0/1156455543 shutdown_connections 2023-11-25T05:04:37.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 1 -- 172.21.15.7:0/1156455543 wait complete. 2023-11-25T05:04:37.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.938+0000 7fe5c6015640 1 librados: shutdown 2023-11-25T05:04:37.955 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509444 for osd.6 2023-11-25T05:04:37.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.956+0000 7f77ad7fa640 1 -- 172.21.15.7:0/214933219 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f77a40073d0 con 0x7f77a8038d80 2023-11-25T05:04:37.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.956+0000 7f77ad7fa640 1 --2- 172.21.15.7:0/214933219 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f779406c800 0x7f779406ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:37.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.956+0000 7f77affff640 1 --2- 172.21.15.7:0/214933219 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f779406c800 0x7f779406ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:37.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.956+0000 7f77affff640 10 monclient: get_auth_request con 0x7f779406c800 auth_method 0 2023-11-25T05:04:37.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.957+0000 7f77affff640 1 --2- 172.21.15.7:0/214933219 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f779406c800 0x7f779406ecc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f779c000f60 tx=0x7f779c0079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:37.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.957+0000 7f77ad7fa640 10 client.4295.objecter ms_handle_connect 0x7f779406c800 2023-11-25T05:04:37.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.973+0000 7f77ad7fa640 1 -- 172.21.15.7:0/214933219 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f77a4057cf0 con 0x7f77a8038d80 2023-11-25T05:04:37.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.973+0000 7f77ad7fa640 10 client.4295.objecter ms_dispatch 0x7f77b0000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:37.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.973+0000 7f77ad7fa640 3 client.4295.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.973+0000 7f77ad7fa640 3 client.4295.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.974+0000 7f77ad7fa640 20 client.4295.objecter dump_active .. 0 homeless 2023-11-25T05:04:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.974+0000 7f77ad7fa640 1 -- 172.21.15.7:0/214933219 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77a4054030 con 0x7f77a8038d80 2023-11-25T05:04:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.974+0000 7f77ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:37.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:37.974+0000 7f77ad7fa640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:38.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-25T05:04:38.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.152+0000 7f4f9add5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-11-25T05:04:38.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.152+0000 7f4f9add5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-11-25T05:04:38.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.152+0000 7f4f9add5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:38.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.152+0000 7f4f9add5640 1 -- 172.21.15.7:0/3084147837 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f4f94142eb0 con 0x7f4f940a8a30 2023-11-25T05:04:38.161 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-25T05:04:38.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.162+0000 7f6b53abb640 10 monclient: get_monmap_and_config 2023-11-25T05:04:38.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.162+0000 7f6b53abb640 10 monclient: build_initial_monmap 2023-11-25T05:04:38.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.162+0000 7f6b53abb640 10 monclient: monmap: 2023-11-25T05:04:38.164 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:38.164 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.164 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:38.163898+0000 2023-11-25T05:04:38.164 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:38.163898+0000 2023-11-25T05:04:38.165 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:38.165 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.165 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:38.165 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:38.165 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:38.165 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:38.165 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:38.165 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.163+0000 7f6b53abb640 1 Processor -- start 2023-11-25T05:04:38.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.163+0000 7f6b53abb640 1 -- start start 2023-11-25T05:04:38.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.163+0000 7f6b53abb640 10 monclient: init 2023-11-25T05:04:38.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.164+0000 7f6b53abb640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:38.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.164+0000 7f6b53abb640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2023-11-25T05:04:38.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.164+0000 7f6b53abb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b4c0a8a30 0x7f6b4c1424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.164+0000 7f6b53abb640 10 monclient(hunting): picked mon.noname-a con 0x7f6b4c0a8a30 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:38.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.164+0000 7f6b53abb640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6b4c0a8e10 0x7f6b4c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.164+0000 7f6b53abb640 10 monclient(hunting): picked mon.noname-c con 0x7f6b4c0a8e10 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:38.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.164+0000 7f6b53abb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b4c1454a0 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:38.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.164+0000 7f6b53abb640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b4c0aac10 con 0x7f6b4c1454a0 2023-11-25T05:04:38.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.164+0000 7f6b53abb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.164+0000 7f6b53abb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b4c0aaa10 con 0x7f6b4c0a8a30 2023-11-25T05:04:38.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.164+0000 7f6b53abb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.164+0000 7f6b53abb640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b4c0aaef0 con 0x7f6b4c0a8e10 2023-11-25T05:04:38.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.164+0000 7f6b53abb640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:38.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.164+0000 7f6b53abb640 10 monclient(hunting): authenticate will time out at 1404.407662s 2023-11-25T05:04:38.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.175+0000 7f6b51056640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6b4c0a8e10 0x7f6b4c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:38.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.176+0000 7f6b52058640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6b4c1454a0 0x7f6b4c1458f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:52172/0 (socket says 172.21.15.7:52172) 2023-11-25T05:04:38.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.176+0000 7f6b52058640 1 -- 172.21.15.7:0/4087284239 learned_addr learned my addr 172.21.15.7:0/4087284239 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:38.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.176+0000 7f6b51056640 10 monclient(hunting): get_auth_request con 0x7f6b4c0a8e10 auth_method 0 2023-11-25T05:04:38.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.176+0000 7f6b51056640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:38.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.176+0000 7f6b51056640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:38.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.176+0000 7f6b51056640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:38.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.178+0000 7f6b51056640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:38.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.178+0000 7f6b51056640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:38.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.178+0000 7f6b51056640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:38.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.180+0000 7f6b51056640 10 monclient(hunting): handle_auth_done global_id 4348 payload 274 2023-11-25T05:04:38.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.180+0000 7f6b51056640 1 -- 172.21.15.7:0/4087284239 >> v2:172.21.15.7:3300/0 conn(0x7f6b4c0a8a30 msgr2=0x7f6b4c1424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:38.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.180+0000 7f6b51056640 1 --2- 172.21.15.7:0/4087284239 >> v2:172.21.15.7:3300/0 conn(0x7f6b4c0a8a30 0x7f6b4c1424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.180+0000 7f6b51056640 1 -- 172.21.15.7:0/4087284239 >> v1:172.21.15.90:6789/0 conn(0x7f6b4c1454a0 legacy=0x7f6b4c1458f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.180+0000 7f6b51056640 10 monclient: _finish_hunting 0 2023-11-25T05:04:38.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.180+0000 7f6b51056640 1 monclient: found mon.noname-c 2023-11-25T05:04:38.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.180+0000 7f6b51056640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:38.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.180+0000 7f6b51056640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:38.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.180+0000 7f6b51056640 1 -- 172.21.15.7:0/4087284239 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4c0a97f0 con 0x7f6b4c0a8e10 2023-11-25T05:04:38.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.183+0000 7f6b51056640 1 --2- 172.21.15.7:0/4087284239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b4c0a8e10 0x7f6b4c142a20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c0b87d0 tx=0x7f6b3c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ed235705a1d5b78 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:38.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b50855640 1 -- 172.21.15.7:0/4087284239 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c030610 con 0x7f6b4c0a8e10 2023-11-25T05:04:38.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b50855640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b50855640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:38.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b50855640 10 monclient: dump: 2023-11-25T05:04:38.194 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.194 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.194 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.194 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.195 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b50855640 10 monclient: _finish_auth 0 2023-11-25T05:04:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b50855640 10 monclient: _check_auth_tickets 2023-11-25T05:04:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b50855640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b50855640 1 -- 172.21.15.7:0/4087284239 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3c030770 con 0x7f6b4c0a8e10 2023-11-25T05:04:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b50855640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b50855640 1 -- 172.21.15.7:0/4087284239 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c030b00 con 0x7f6b4c0a8e10 2023-11-25T05:04:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b50855640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b50855640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b50855640 10 monclient: dump: 2023-11-25T05:04:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.197 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b53abb640 5 monclient: authenticate success, global_id 4348 2023-11-25T05:04:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b53abb640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b53abb640 4 set_mon_vals no callback set 2023-11-25T05:04:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b53abb640 10 monclient: shutdown 2023-11-25T05:04:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b53abb640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b53abb640 1 -- 172.21.15.7:0/4087284239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b4c0a8e10 msgr2=0x7f6b4c142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b53abb640 1 --2- 172.21.15.7:0/4087284239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b4c0a8e10 0x7f6b4c142a20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c0b87d0 tx=0x7f6b3c02b960 comp rx=0 tx=0).stop 2023-11-25T05:04:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b53abb640 1 -- 172.21.15.7:0/4087284239 shutdown_connections 2023-11-25T05:04:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b53abb640 1 --2- 172.21.15.7:0/4087284239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b4c0a8e10 0x7f6b4c142a20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b53abb640 1 --2- 172.21.15.7:0/4087284239 >> v2:172.21.15.7:3300/0 conn(0x7f6b4c0a8a30 0x7f6b4c1424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.188+0000 7f6b53abb640 1 -- 172.21.15.7:0/4087284239 >> 172.21.15.7:0/4087284239 conn(0x7f6b4c061a30 msgr2=0x7f6b4c0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.189+0000 7f6b53abb640 1 -- 172.21.15.7:0/4087284239 shutdown_connections 2023-11-25T05:04:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.189+0000 7f6b53abb640 1 -- 172.21.15.7:0/4087284239 wait complete. 2023-11-25T05:04:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.190+0000 7f6b53abb640 10 monclient: build_initial_monmap 2023-11-25T05:04:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.190+0000 7f6b53abb640 10 monclient: monmap: 2023-11-25T05:04:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:38.199 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:38.199 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.190+0000 7f6b53abb640 1 librados: starting msgr at 2023-11-25T05:04:38.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.190+0000 7f6b53abb640 1 librados: starting objecter 2023-11-25T05:04:38.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.190+0000 7f6b53abb640 1 Processor -- start 2023-11-25T05:04:38.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.190+0000 7f6b53abb640 1 -- start start 2023-11-25T05:04:38.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.190+0000 7f6b53abb640 1 librados: setting wanted keys 2023-11-25T05:04:38.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.190+0000 7f6b53abb640 1 librados: calling monclient init 2023-11-25T05:04:38.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.190+0000 7f6b53abb640 10 monclient: init 2023-11-25T05:04:38.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:38.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-25T05:04:38.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b4c0a8a30 0x7f6b4c13ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 10 monclient(hunting): picked mon.noname-c con 0x7f6b4c0a8a30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:38.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b4c0a8e10 0x7f6b4c1debd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 10 monclient(hunting): picked mon.noname-a con 0x7f6b4c0a8e10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:38.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b4c1454a0 0x7f6b4c1df350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 10 monclient(hunting): picked mon.noname-b con 0x7f6b4c1454a0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:38.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c1df890 con 0x7f6b4c0a8e10 2023-11-25T05:04:38.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c06f830 con 0x7f6b4c1454a0 2023-11-25T05:04:38.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c071820 con 0x7f6b4c0a8a30 2023-11-25T05:04:38.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b53abb640 10 monclient(hunting): authenticate will time out at 1404.434251s 2023-11-25T05:04:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.191+0000 7f6b52058640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b4c1454a0 0x7f6b4c1df350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.192+0000 7f6b52058640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b4c1454a0 0x7f6b4c1df350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:35186/0 (socket says 172.21.15.7:35186) 2023-11-25T05:04:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.192+0000 7f6b52058640 1 -- 172.21.15.7:0/2769942322 learned_addr learned my addr 172.21.15.7:0/2769942322 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.192+0000 7f6b52058640 10 monclient(hunting): get_auth_request con 0x7f6b4c1454a0 auth_method 0 2023-11-25T05:04:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.192+0000 7f6b52058640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.192+0000 7f6b52058640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.192+0000 7f6b52058640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.192+0000 7f6b52058640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.192+0000 7f6b52058640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.192+0000 7f6b52058640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:38.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-25T05:04:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.195+0000 7f6b52058640 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2023-11-25T05:04:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.195+0000 7f6b52058640 1 -- 172.21.15.7:0/2769942322 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b4c0a8a30 msgr2=0x7f6b4c13ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.195+0000 7f6b52058640 1 --2- 172.21.15.7:0/2769942322 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b4c0a8a30 0x7f6b4c13ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.195+0000 7f6b52058640 1 -- 172.21.15.7:0/2769942322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b4c0a8e10 msgr2=0x7f6b4c1debd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.195+0000 7f6b52058640 1 --2- 172.21.15.7:0/2769942322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b4c0a8e10 0x7f6b4c1debd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.195+0000 7f6b52058640 10 monclient: _finish_hunting 0 2023-11-25T05:04:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.195+0000 7f6b52058640 1 monclient: found mon.noname-b 2023-11-25T05:04:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.195+0000 7f6b52058640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.195+0000 7f6b52058640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.195+0000 7f6b52058640 1 -- 172.21.15.7:0/2769942322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c009400 con 0x7f6b4c1454a0 2023-11-25T05:04:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.200+0000 7f6b52058640 1 --2- 172.21.15.7:0/2769942322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b4c1454a0 0x7f6b4c1df350 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400ca80 tx=0x7f6b4400cf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:38.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.200+0000 7f6b427fc640 10 client.?.objecter ms_handle_connect 0x7f6b4c1454a0 2023-11-25T05:04:38.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.200+0000 7f6b427fc640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b427fc640 1 -- 172.21.15.7:0/2769942322 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4400ac30 con 0x7f6b4c1454a0 2023-11-25T05:04:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b427fc640 10 monclient: dump: 2023-11-25T05:04:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.208 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.208 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b427fc640 10 monclient: _finish_auth 0 2023-11-25T05:04:38.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b427fc640 10 monclient: _check_auth_tickets 2023-11-25T05:04:38.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:38.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b427fc640 1 -- 172.21.15.7:0/2769942322 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4400ad90 con 0x7f6b4c1454a0 2023-11-25T05:04:38.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b427fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:38.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b427fc640 1 -- 172.21.15.7:0/2769942322 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b44012600 con 0x7f6b4c1454a0 2023-11-25T05:04:38.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:38.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b427fc640 10 monclient: dump: 2023-11-25T05:04:38.209 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.209 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.209 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.209 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.209 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.209 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.209 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.209 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.209 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.209 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b43fff640 4 set_mon_vals no callback set 2023-11-25T05:04:38.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b53abb640 5 monclient: authenticate success, global_id 4351 2023-11-25T05:04:38.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b53abb640 10 monclient: _renew_subs 2023-11-25T05:04:38.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b53abb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:38.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b53abb640 1 -- 172.21.15.7:0/2769942322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4c06dac0 con 0x7f6b4c1454a0 2023-11-25T05:04:38.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b53abb640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:38.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b53abb640 10 monclient: _renew_subs 2023-11-25T05:04:38.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.204+0000 7f6b53abb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:38.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.205+0000 7f6b53abb640 1 -- 172.21.15.7:0/2769942322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4c06def0 con 0x7f6b4c1454a0 2023-11-25T05:04:38.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.205+0000 7f6b53abb640 1 librados: init done 2023-11-25T05:04:38.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.205+0000 7f77b5f6c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-11-25T05:04:38.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.205+0000 7f77b5f6c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-11-25T05:04:38.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.205+0000 7f77b5f6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:38.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.205+0000 7f77b5f6c640 1 -- 172.21.15.7:0/214933219 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f77a8008600 con 0x7f77a8038d80 2023-11-25T05:04:38.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.205+0000 7f6b53abb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:38.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.205+0000 7f6b53abb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:38.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.205+0000 7f6b53abb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:38.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.205+0000 7f6b53abb640 1 -- 172.21.15.7:0/2769942322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4c067ac0 con 0x7f6b4c1454a0 2023-11-25T05:04:38.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.228+0000 7f6b427fc640 1 -- 172.21.15.7:0/2769942322 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6b44007070 con 0x7f6b4c1454a0 2023-11-25T05:04:38.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.229+0000 7f6b427fc640 1 --2- 172.21.15.7:0/2769942322 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6b3006c820 0x7f6b3006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.229+0000 7f6b51857640 1 --2- 172.21.15.7:0/2769942322 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6b3006c820 0x7f6b3006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:38.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.229+0000 7f6b51857640 10 monclient: get_auth_request con 0x7f6b3006c820 auth_method 0 2023-11-25T05:04:38.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.229+0000 7f6b51857640 1 --2- 172.21.15.7:0/2769942322 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6b3006c820 0x7f6b3006ece0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009710 tx=0x7f6b48006c90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:38.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.229+0000 7f6b427fc640 10 client.4351.objecter ms_handle_connect 0x7f6b3006c820 2023-11-25T05:04:38.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.235+0000 7f4f917fa640 1 -- 172.21.15.7:0/3084147837 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4f94142eb0 con 0x7f4f940a8a30 2023-11-25T05:04:38.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.235+0000 7f4f917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-11-25T05:04:38.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.235+0000 7f4f917fa640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:38.238 INFO:teuthology.orchestra.run.smithi007.stdout:68719476740 2023-11-25T05:04:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.237+0000 7f4f9add5640 10 librados: watch_flush enter 2023-11-25T05:04:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.237+0000 7f4f9add5640 10 librados: watch_flush exit 2023-11-25T05:04:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.237+0000 7f4f9add5640 20 client.4289.objecter shutdown clearing up homeless session... 2023-11-25T05:04:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.237+0000 7f4f9add5640 10 client.4289.objecter successfully canceled tick 2023-11-25T05:04:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.237+0000 7f4f9add5640 1 -- 172.21.15.7:0/3084147837 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f4f8006c800 msgr2=0x7f4f8006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.237+0000 7f4f9add5640 1 --2- 172.21.15.7:0/3084147837 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f4f8006c800 0x7f4f8006ecc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4f941d4c70 tx=0x7f4f88007d10 comp rx=0 tx=0).stop 2023-11-25T05:04:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.237+0000 7f4f9add5640 10 monclient: shutdown 2023-11-25T05:04:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.237+0000 7f4f9add5640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.237+0000 7f4f9add5640 1 -- 172.21.15.7:0/3084147837 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f940a8a30 msgr2=0x7f4f940aed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.238+0000 7f4f9add5640 1 --2- 172.21.15.7:0/3084147837 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f940a8a30 0x7f4f940aed50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4f8402b930 tx=0x7f4f840046c0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.238+0000 7f4f9add5640 1 -- 172.21.15.7:0/3084147837 shutdown_connections 2023-11-25T05:04:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.238+0000 7f4f9add5640 1 --2- 172.21.15.7:0/3084147837 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f4f8006c800 0x7f4f8006ecc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.238+0000 7f4f9add5640 1 --2- 172.21.15.7:0/3084147837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f94142a20 0x7f4f9413b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.238+0000 7f4f9add5640 1 --2- 172.21.15.7:0/3084147837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f940a8e10 0x7f4f940af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.238+0000 7f4f9add5640 1 --2- 172.21.15.7:0/3084147837 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f940a8a30 0x7f4f940aed50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.238+0000 7f4f9add5640 1 -- 172.21.15.7:0/3084147837 >> 172.21.15.7:0/3084147837 conn(0x7f4f94061a30 msgr2=0x7f4f9413cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.238+0000 7f4f9add5640 1 -- 172.21.15.7:0/3084147837 shutdown_connections 2023-11-25T05:04:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.238+0000 7f4f9add5640 1 -- 172.21.15.7:0/3084147837 wait complete. 2023-11-25T05:04:38.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.238+0000 7f4f9add5640 1 librados: shutdown 2023-11-25T05:04:38.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.255+0000 7f77ad7fa640 1 -- 172.21.15.7:0/214933219 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f77a4017090 con 0x7f77a8038d80 2023-11-25T05:04:38.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.255+0000 7f77ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-11-25T05:04:38.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.255+0000 7f77ad7fa640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:38.262 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.2 2023-11-25T05:04:38.262 INFO:teuthology.orchestra.run.smithi007.stdout:55834574851 2023-11-25T05:04:38.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 10 librados: watch_flush enter 2023-11-25T05:04:38.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 10 librados: watch_flush exit 2023-11-25T05:04:38.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 20 client.4295.objecter shutdown clearing up homeless session... 2023-11-25T05:04:38.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 10 client.4295.objecter successfully canceled tick 2023-11-25T05:04:38.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 1 -- 172.21.15.7:0/214933219 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f779406c800 msgr2=0x7f779406ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 1 --2- 172.21.15.7:0/214933219 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f779406c800 0x7f779406ecc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f779c000f60 tx=0x7f779c0079b0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 10 monclient: shutdown 2023-11-25T05:04:38.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:38.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 1 -- 172.21.15.7:0/214933219 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f77a8038d80 msgr2=0x7f77a802fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 1 --2- 172.21.15.7:0/214933219 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f77a8038d80 0x7f77a802fc00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f77a400c4a0 tx=0x7f77a400c960 comp rx=0 tx=0).stop 2023-11-25T05:04:38.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 1 -- 172.21.15.7:0/214933219 shutdown_connections 2023-11-25T05:04:38.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 1 --2- 172.21.15.7:0/214933219 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f779406c800 0x7f779406ecc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 1 --2- 172.21.15.7:0/214933219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a80c2c30 0x7f77a80c30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 1 --2- 172.21.15.7:0/214933219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a80c8a30 0x7f77a8160940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 1 --2- 172.21.15.7:0/214933219 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f77a8038d80 0x7f77a802fc00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 1 -- 172.21.15.7:0/214933219 >> 172.21.15.7:0/214933219 conn(0x7f77a80065e0 msgr2=0x7f77a80c4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:38.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 1 -- 172.21.15.7:0/214933219 shutdown_connections 2023-11-25T05:04:38.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.257+0000 7f77b5f6c640 1 -- 172.21.15.7:0/214933219 wait complete. 2023-11-25T05:04:38.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.258+0000 7f77b5f6c640 1 librados: shutdown 2023-11-25T05:04:38.272 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.7 2023-11-25T05:04:38.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.291+0000 7ffacf1a9640 10 monclient: get_monmap_and_config 2023-11-25T05:04:38.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.291+0000 7ffacf1a9640 10 monclient: build_initial_monmap 2023-11-25T05:04:38.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.291+0000 7ffacf1a9640 10 monclient: monmap: 2023-11-25T05:04:38.293 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:38.294 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.294 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:38.292761+0000 2023-11-25T05:04:38.294 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:38.292761+0000 2023-11-25T05:04:38.294 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:38.294 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.294 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:38.294 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:38.294 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:38.294 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:38.294 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:38.294 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.292+0000 7ffacf1a9640 1 Processor -- start 2023-11-25T05:04:38.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.292+0000 7ffacf1a9640 1 -- start start 2023-11-25T05:04:38.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.292+0000 7ffacf1a9640 10 monclient: init 2023-11-25T05:04:38.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.293+0000 7ffacf1a9640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:38.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.293+0000 7ffacf1a9640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-11-25T05:04:38.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.293+0000 7ffacf1a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffac806ccd0 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:38.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.293+0000 7ffacf1a9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffac806c0e0 0x7ffac8071700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.293+0000 7ffacf1a9640 10 monclient(hunting): picked mon.noname-a con 0x7ffac806c0e0 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:38.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.293+0000 7ffacf1a9640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ffac806d0b0 0x7ffac8071c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.293+0000 7ffacf1a9640 10 monclient(hunting): picked mon.noname-c con 0x7ffac806d0b0 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:38.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.293+0000 7ffacf1a9640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac80afb80 con 0x7ffac806ccd0 2023-11-25T05:04:38.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.293+0000 7ffacf1a9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.293+0000 7ffacf1a9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac80b0010 con 0x7ffac806c0e0 2023-11-25T05:04:38.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.293+0000 7ffacf1a9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.293+0000 7ffacf1a9640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac8071070 con 0x7ffac806d0b0 2023-11-25T05:04:38.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.293+0000 7ffacf1a9640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:38.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.293+0000 7ffacf1a9640 10 monclient(hunting): authenticate will time out at 1404.536149s 2023-11-25T05:04:38.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.294+0000 7ffac7fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffac806c0e0 0x7ffac8071700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:38.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.294+0000 7ffac7fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffac806c0e0 0x7ffac8071700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58354/0 (socket says 172.21.15.7:58354) 2023-11-25T05:04:38.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.294+0000 7ffac7fff640 1 -- 172.21.15.7:0/3982833990 learned_addr learned my addr 172.21.15.7:0/3982833990 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:38.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.294+0000 7ffac7fff640 10 monclient(hunting): get_auth_request con 0x7ffac806c0e0 auth_method 0 2023-11-25T05:04:38.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.294+0000 7ffac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:38.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.294+0000 7ffac7fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:38.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.294+0000 7ffac7fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:38.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.294+0000 7ffac7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:38.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.294+0000 7ffac7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:38.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.294+0000 7ffac7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:38.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.296+0000 7ffac7fff640 10 monclient(hunting): handle_auth_done global_id 4320 payload 274 2023-11-25T05:04:38.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.296+0000 7ffac7fff640 1 -- 172.21.15.7:0/3982833990 >> v2:172.21.15.90:3300/0 conn(0x7ffac806d0b0 msgr2=0x7ffac8071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:38.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.296+0000 7ffac7fff640 1 --2- 172.21.15.7:0/3982833990 >> v2:172.21.15.90:3300/0 conn(0x7ffac806d0b0 0x7ffac8071c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.296+0000 7ffac7fff640 1 -- 172.21.15.7:0/3982833990 >> v1:172.21.15.90:6789/0 conn(0x7ffac806ccd0 legacy=0x7ffac813f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:38.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.296+0000 7ffac7fff640 10 monclient: _finish_hunting 0 2023-11-25T05:04:38.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.296+0000 7ffac7fff640 1 monclient: found mon.noname-a 2023-11-25T05:04:38.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.296+0000 7ffac7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:38.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.296+0000 7ffac7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:38.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.296+0000 7ffac7fff640 1 -- 172.21.15.7:0/3982833990 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac80712e0 con 0x7ffac806c0e0 2023-11-25T05:04:38.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.298+0000 7ffac7fff640 1 --2- 172.21.15.7:0/3982833990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac806c0e0 0x7ffac8071700 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffac81463c0 tx=0x7ffab002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6b41d2e5b3601d server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:38.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffac77fe640 1 -- 172.21.15.7:0/3982833990 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab00317e0 con 0x7ffac806c0e0 2023-11-25T05:04:38.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffac77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffac77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:38.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffac77fe640 10 monclient: dump: 2023-11-25T05:04:38.306 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.306 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.306 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.307 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.307 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.307 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.307 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.307 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.307 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.307 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffac77fe640 10 monclient: _finish_auth 0 2023-11-25T05:04:38.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffac77fe640 10 monclient: _check_auth_tickets 2023-11-25T05:04:38.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffac77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:38.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffac77fe640 1 -- 172.21.15.7:0/3982833990 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab0031940 con 0x7ffac806c0e0 2023-11-25T05:04:38.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffac77fe640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:38.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffac77fe640 1 -- 172.21.15.7:0/3982833990 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0031cd0 con 0x7ffac806c0e0 2023-11-25T05:04:38.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffac77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffac77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:38.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffac77fe640 10 monclient: dump: 2023-11-25T05:04:38.308 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.308 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.309 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.309 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.309 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.309 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.309 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.309 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.309 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.309 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffacf1a9640 5 monclient: authenticate success, global_id 4320 2023-11-25T05:04:38.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffacf1a9640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:38.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.302+0000 7ffacf1a9640 4 set_mon_vals no callback set 2023-11-25T05:04:38.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.303+0000 7ffacf1a9640 10 monclient: shutdown 2023-11-25T05:04:38.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.303+0000 7ffacf1a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:38.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.303+0000 7ffacf1a9640 1 -- 172.21.15.7:0/3982833990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac806c0e0 msgr2=0x7ffac8071700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.303+0000 7ffacf1a9640 1 --2- 172.21.15.7:0/3982833990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac806c0e0 0x7ffac8071700 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffac81463c0 tx=0x7ffab002b880 comp rx=0 tx=0).stop 2023-11-25T05:04:38.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.303+0000 7ffacf1a9640 1 -- 172.21.15.7:0/3982833990 shutdown_connections 2023-11-25T05:04:38.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.303+0000 7ffacf1a9640 1 --2- 172.21.15.7:0/3982833990 >> v2:172.21.15.90:3300/0 conn(0x7ffac806d0b0 0x7ffac8071c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.303+0000 7ffacf1a9640 1 --2- 172.21.15.7:0/3982833990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac806c0e0 0x7ffac8071700 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.303+0000 7ffacf1a9640 1 -- 172.21.15.7:0/3982833990 >> 172.21.15.7:0/3982833990 conn(0x7ffac80b5ca0 msgr2=0x7ffac80b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:38.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.303+0000 7ffacf1a9640 1 -- 172.21.15.7:0/3982833990 shutdown_connections 2023-11-25T05:04:38.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.303+0000 7ffacf1a9640 1 -- 172.21.15.7:0/3982833990 wait complete. 2023-11-25T05:04:38.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.304+0000 7ffacf1a9640 10 monclient: build_initial_monmap 2023-11-25T05:04:38.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.304+0000 7ffacf1a9640 10 monclient: monmap: 2023-11-25T05:04:38.311 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:38.311 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:38.311 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:38.311 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:38.311 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:38.311 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.311 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:38.311 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:38.311 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:38.311 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.304+0000 7ffacf1a9640 1 librados: starting msgr at 2023-11-25T05:04:38.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.304+0000 7ffacf1a9640 1 librados: starting objecter 2023-11-25T05:04:38.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.304+0000 7ffacf1a9640 1 Processor -- start 2023-11-25T05:04:38.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.304+0000 7ffacf1a9640 1 -- start start 2023-11-25T05:04:38.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.304+0000 7ffacf1a9640 1 librados: setting wanted keys 2023-11-25T05:04:38.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.304+0000 7ffacf1a9640 1 librados: calling monclient init 2023-11-25T05:04:38.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.304+0000 7ffacf1a9640 10 monclient: init 2023-11-25T05:04:38.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:38.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-25T05:04:38.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac806c0e0 0x7ffac813b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 10 monclient(hunting): picked mon.noname-a con 0x7ffac806c0e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:38.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffac806ccd0 0x7ffac8137d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 10 monclient(hunting): picked mon.noname-b con 0x7ffac806ccd0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:38.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffac806d0b0 0x7ffac8139720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 10 monclient(hunting): picked mon.noname-c con 0x7ffac806d0b0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:38.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac80ad3a0 con 0x7ffac806c0e0 2023-11-25T05:04:38.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac80ad680 con 0x7ffac806ccd0 2023-11-25T05:04:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac813c000 con 0x7ffac806d0b0 2023-11-25T05:04:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.305+0000 7ffacf1a9640 10 monclient(hunting): authenticate will time out at 1404.548415s 2023-11-25T05:04:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.306+0000 7ffaccf45640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac806c0e0 0x7ffac813b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.306+0000 7ffacd746640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffac806d0b0 0x7ffac8139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.306+0000 7ffaccf45640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac806c0e0 0x7ffac813b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58356/0 (socket says 172.21.15.7:58356) 2023-11-25T05:04:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.306+0000 7ffaccf45640 1 -- 172.21.15.7:0/420940907 learned_addr learned my addr 172.21.15.7:0/420940907 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.306+0000 7ffaccf45640 10 monclient(hunting): get_auth_request con 0x7ffac806c0e0 auth_method 0 2023-11-25T05:04:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.306+0000 7ffaccf45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.306+0000 7ffaccf45640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.306+0000 7ffaccf45640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:38.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.306+0000 7ffacd746640 10 monclient(hunting): get_auth_request con 0x7ffac806d0b0 auth_method 0 2023-11-25T05:04:38.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.306+0000 7ffacd746640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:38.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.306+0000 7ffacd746640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:38.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.306+0000 7ffacd746640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:38.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.306+0000 7ffacd746640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:38.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.306+0000 7ffacd746640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:38.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.307+0000 7ffacd746640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:38.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.307+0000 7ffaccf45640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:38.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.307+0000 7ffaccf45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:38.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.307+0000 7ffaccf45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:38.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.309+0000 7ffacd746640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2023-11-25T05:04:38.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.310+0000 7ffacd746640 1 -- 172.21.15.7:0/420940907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffac806ccd0 msgr2=0x7ffac8137d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.310+0000 7ffacd746640 1 --2- 172.21.15.7:0/420940907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffac806ccd0 0x7ffac8137d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.310+0000 7ffacd746640 1 -- 172.21.15.7:0/420940907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac806c0e0 msgr2=0x7ffac813b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.310+0000 7ffacd746640 1 --2- 172.21.15.7:0/420940907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac806c0e0 0x7ffac813b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.310+0000 7ffacd746640 10 monclient: _finish_hunting 0 2023-11-25T05:04:38.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.310+0000 7ffacd746640 1 monclient: found mon.noname-c 2023-11-25T05:04:38.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.310+0000 7ffacd746640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:38.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.310+0000 7ffacd746640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:04:38.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.310+0000 7ffacd746640 1 -- 172.21.15.7:0/420940907 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab00094d0 con 0x7ffac806d0b0 2023-11-25T05:04:38.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.312+0000 7ffacd746640 1 --2- 172.21.15.7:0/420940907 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffac806d0b0 0x7ffac8139720 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffac000be20 tx=0x7ffac000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:38.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.312+0000 7ffac57fa640 10 client.?.objecter ms_handle_connect 0x7ffac806d0b0 2023-11-25T05:04:38.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.312+0000 7ffac57fa640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:38.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffac57fa640 1 -- 172.21.15.7:0/420940907 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac000ce70 con 0x7ffac806d0b0 2023-11-25T05:04:38.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffac57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffac57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:38.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffac57fa640 10 monclient: dump: 2023-11-25T05:04:38.320 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.320 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.320 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.321 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffac57fa640 10 monclient: _finish_auth 0 2023-11-25T05:04:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffac57fa640 10 monclient: _check_auth_tickets 2023-11-25T05:04:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffac57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffac57fa640 1 -- 172.21.15.7:0/420940907 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac0014440 con 0x7ffac806d0b0 2023-11-25T05:04:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffacf1a9640 5 monclient: authenticate success, global_id 4298 2023-11-25T05:04:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffacf1a9640 10 monclient: _renew_subs 2023-11-25T05:04:38.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffacf1a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:38.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffacf1a9640 1 -- 172.21.15.7:0/420940907 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac8138240 con 0x7ffac806d0b0 2023-11-25T05:04:38.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffac57fa640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:38.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffac57fa640 1 -- 172.21.15.7:0/420940907 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac0014810 con 0x7ffac806d0b0 2023-11-25T05:04:38.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffac57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffac57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:38.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffac57fa640 10 monclient: dump: 2023-11-25T05:04:38.322 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.322 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.322 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.322 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.322 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.323 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.323 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.323 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.323 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.323 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffac6ffd640 4 set_mon_vals no callback set 2023-11-25T05:04:38.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffacf1a9640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:38.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffacf1a9640 10 monclient: _renew_subs 2023-11-25T05:04:38.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffacf1a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:38.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffacf1a9640 1 -- 172.21.15.7:0/420940907 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac8139c60 con 0x7ffac806d0b0 2023-11-25T05:04:38.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.317+0000 7ffacf1a9640 1 librados: init done 2023-11-25T05:04:38.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.318+0000 7ffacf1a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:38.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.318+0000 7ffacf1a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:38.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.318+0000 7ffacf1a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:38.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.318+0000 7ffacf1a9640 1 -- 172.21.15.7:0/420940907 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac813be60 con 0x7ffac806d0b0 2023-11-25T05:04:38.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.327+0000 7fba71b01640 10 monclient: get_monmap_and_config 2023-11-25T05:04:38.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.327+0000 7fba71b01640 10 monclient: build_initial_monmap 2023-11-25T05:04:38.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.327+0000 7fba71b01640 10 monclient: monmap: 2023-11-25T05:04:38.329 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:38.329 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.330 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:38.328899+0000 2023-11-25T05:04:38.330 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:38.328899+0000 2023-11-25T05:04:38.330 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:38.330 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.330 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:38.330 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:38.330 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:38.330 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:38.330 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:38.330 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.328+0000 7fba71b01640 1 Processor -- start 2023-11-25T05:04:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.328+0000 7fba71b01640 1 -- start start 2023-11-25T05:04:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.328+0000 7fba71b01640 10 monclient: init 2023-11-25T05:04:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.329+0000 7fba71b01640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.329+0000 7fba71b01640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-11-25T05:04:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.329+0000 7fba71b01640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba6c0a8a30 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.329+0000 7fba71b01640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fba6c142bf0 0x7fba6c13ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.329+0000 7fba71b01640 10 monclient(hunting): picked mon.noname-c con 0x7fba6c142bf0 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.329+0000 7fba71b01640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fba6c0a8e10 0x7fba6c13f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.329+0000 7fba71b01640 10 monclient(hunting): picked mon.noname-b con 0x7fba6c0a8e10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.329+0000 7fba71b01640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c0aac10 con 0x7fba6c0a8a30 2023-11-25T05:04:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.329+0000 7fba71b01640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.329+0000 7fba71b01640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba6c0aaa10 con 0x7fba6c142bf0 2023-11-25T05:04:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.329+0000 7fba71b01640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.329+0000 7fba71b01640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c0aaef0 con 0x7fba6c0a8e10 2023-11-25T05:04:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.329+0000 7fba71b01640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.329+0000 7fba71b01640 10 monclient(hunting): authenticate will time out at 1404.572238s 2023-11-25T05:04:38.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.339+0000 7ffac57fa640 1 -- 172.21.15.7:0/420940907 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ffac001d3b0 con 0x7ffac806d0b0 2023-11-25T05:04:38.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.339+0000 7ffac57fa640 1 --2- 172.21.15.7:0/420940907 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ffaa806c820 0x7ffaa806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.340+0000 7ffaccf45640 1 --2- 172.21.15.7:0/420940907 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ffaa806c820 0x7ffaa806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:38.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.340+0000 7ffaccf45640 10 monclient: get_auth_request con 0x7ffaa806c820 auth_method 0 2023-11-25T05:04:38.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.340+0000 7ffaccf45640 1 --2- 172.21.15.7:0/420940907 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ffaa806c820 0x7ffaa806ece0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffab80018b0 tx=0x7ffab8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:38.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.340+0000 7ffac57fa640 10 client.4298.objecter ms_handle_connect 0x7ffaa806c820 2023-11-25T05:04:38.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.343+0000 7ffac57fa640 1 -- 172.21.15.7:0/420940907 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ffac0058870 con 0x7ffac806d0b0 2023-11-25T05:04:38.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.343+0000 7ffac57fa640 10 client.4298.objecter ms_dispatch 0x7ffac8000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:38.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.343+0000 7ffac57fa640 3 client.4298.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:38.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.343+0000 7ffac57fa640 3 client.4298.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:38.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.343+0000 7ffac57fa640 20 client.4298.objecter dump_active .. 0 homeless 2023-11-25T05:04:38.374 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-25T05:04:38.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.457+0000 7f6b427fc640 1 -- 172.21.15.7:0/2769942322 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6b44058780 con 0x7f6b4c1454a0 2023-11-25T05:04:38.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.457+0000 7f6b427fc640 10 client.4351.objecter ms_dispatch 0x7f6b4c000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:38.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.457+0000 7f6b427fc640 3 client.4351.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:38.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.457+0000 7f6b427fc640 3 client.4351.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:38.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.457+0000 7f6b427fc640 20 client.4351.objecter dump_active .. 0 homeless 2023-11-25T05:04:38.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.468+0000 7f6b427fc640 1 -- 172.21.15.7:0/2769942322 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b44054ac0 con 0x7f6b4c1454a0 2023-11-25T05:04:38.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.468+0000 7f6b427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:38.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.468+0000 7f6b427fc640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:38.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.472+0000 7fba6affd640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fba6c142bf0 0x7fba6c13ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:38.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.472+0000 7fba6affd640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fba6c142bf0 0x7fba6c13ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:35216/0 (socket says 172.21.15.7:35216) 2023-11-25T05:04:38.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.472+0000 7fba6affd640 1 -- 172.21.15.7:0/515766973 learned_addr learned my addr 172.21.15.7:0/515766973 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:38.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.472+0000 7fba6affd640 10 monclient(hunting): get_auth_request con 0x7fba6c142bf0 auth_method 0 2023-11-25T05:04:38.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.472+0000 7fba6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:38.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.472+0000 7fba6affd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:38.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.472+0000 7fba6affd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:38.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.473+0000 7fba6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:38.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.473+0000 7fba6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:38.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.473+0000 7fba6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:38.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.475+0000 7fba6affd640 10 monclient(hunting): handle_auth_done global_id 4354 payload 274 2023-11-25T05:04:38.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.475+0000 7fba6affd640 1 -- 172.21.15.7:0/515766973 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fba6c0a8e10 msgr2=0x7fba6c13f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:38.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.475+0000 7fba6affd640 1 --2- 172.21.15.7:0/515766973 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fba6c0a8e10 0x7fba6c13f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.475+0000 7fba6affd640 1 -- 172.21.15.7:0/515766973 >> v1:172.21.15.90:6789/0 conn(0x7fba6c0a8a30 legacy=0x7fba6c1424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:38.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.475+0000 7fba6affd640 10 monclient: _finish_hunting 0 2023-11-25T05:04:38.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.475+0000 7fba6affd640 1 monclient: found mon.noname-c 2023-11-25T05:04:38.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.475+0000 7fba6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:38.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.475+0000 7fba6affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:38.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.475+0000 7fba6affd640 1 -- 172.21.15.7:0/515766973 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c0a97f0 con 0x7fba6c142bf0 2023-11-25T05:04:38.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.480+0000 7fba6affd640 1 --2- 172.21.15.7:0/515766973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba6c142bf0 0x7fba6c13ecf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009870 tx=0x7fba5c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fda9b35529d66d8c server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:38.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.481+0000 7f4c3b5e9640 10 monclient: get_monmap_and_config 2023-11-25T05:04:38.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.481+0000 7f4c3b5e9640 10 monclient: build_initial_monmap 2023-11-25T05:04:38.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.481+0000 7f4c3b5e9640 10 monclient: monmap: 2023-11-25T05:04:38.484 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:38.484 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.484 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:38.482022+0000 2023-11-25T05:04:38.484 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:38.482022+0000 2023-11-25T05:04:38.484 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:38.485 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.485 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:38.485 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:38.485 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:38.485 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:38.485 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:38.485 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.481+0000 7f4c3b5e9640 1 Processor -- start 2023-11-25T05:04:38.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.481+0000 7f4c3b5e9640 1 -- start start 2023-11-25T05:04:38.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.481+0000 7f4c3b5e9640 10 monclient: init 2023-11-25T05:04:38.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.482+0000 7f4c3b5e9640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:38.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.482+0000 7f4c3b5e9640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-11-25T05:04:38.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.482+0000 7f4c3b5e9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c340a8910 0x7f4c3413ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.482+0000 7f4c3b5e9640 10 monclient(hunting): picked mon.noname-b con 0x7f4c340a8910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:38.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.482+0000 7f4c3b5e9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c340a9b70 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:38.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.482+0000 7f4c3b5e9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c34141f90 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:38.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.482+0000 7f4c3b5e9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c340a8cf0 con 0x7f4c34141f90 2023-11-25T05:04:38.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.482+0000 7f4c3b5e9640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c340aabd0 con 0x7f4c340a9b70 2023-11-25T05:04:38.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.482+0000 7f4c3b5e9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.482+0000 7f4c3b5e9640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c340aaeb0 con 0x7f4c340a8910 2023-11-25T05:04:38.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.482+0000 7f4c3b5e9640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:38.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.482+0000 7f4c3b5e9640 10 monclient(hunting): authenticate will time out at 1404.725531s 2023-11-25T05:04:38.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba6a7fc640 1 -- 172.21.15.7:0/515766973 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5c031640 con 0x7fba6c142bf0 2023-11-25T05:04:38.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba6a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:38.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba6a7fc640 10 monclient: dump: 2023-11-25T05:04:38.487 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.487 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.487 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.487 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.487 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.487 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.487 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.487 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.488 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba6a7fc640 10 monclient: _finish_auth 0 2023-11-25T05:04:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba6a7fc640 10 monclient: _check_auth_tickets 2023-11-25T05:04:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba6a7fc640 1 -- 172.21.15.7:0/515766973 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5c0317a0 con 0x7fba6c142bf0 2023-11-25T05:04:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba6a7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba6a7fc640 1 -- 172.21.15.7:0/515766973 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5c031b30 con 0x7fba6c142bf0 2023-11-25T05:04:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba71b01640 5 monclient: authenticate success, global_id 4354 2023-11-25T05:04:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba71b01640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba71b01640 4 set_mon_vals no callback set 2023-11-25T05:04:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7fba6a7fc640 10 monclient: dump: 2023-11-25T05:04:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.489 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.489 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7f4c38b84640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4c340a9b70 0x7f4c3413f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:52202/0 (socket says 172.21.15.7:52202) 2023-11-25T05:04:38.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.485+0000 7f4c38b84640 1 -- 172.21.15.7:0/1542095784 learned_addr learned my addr 172.21.15.7:0/1542095784 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:38.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.486+0000 7fba71b01640 10 monclient: shutdown 2023-11-25T05:04:38.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.486+0000 7fba71b01640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:38.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.486+0000 7fba71b01640 1 -- 172.21.15.7:0/515766973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba6c142bf0 msgr2=0x7fba6c13ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.486+0000 7fba71b01640 1 --2- 172.21.15.7:0/515766973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba6c142bf0 0x7fba6c13ecf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009870 tx=0x7fba5c02b8f0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.487+0000 7fba71b01640 1 -- 172.21.15.7:0/515766973 shutdown_connections 2023-11-25T05:04:38.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.487+0000 7fba71b01640 1 --2- 172.21.15.7:0/515766973 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fba6c0a8e10 0x7fba6c13f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.487+0000 7fba71b01640 1 --2- 172.21.15.7:0/515766973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba6c142bf0 0x7fba6c13ecf0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.487+0000 7fba71b01640 1 -- 172.21.15.7:0/515766973 >> 172.21.15.7:0/515766973 conn(0x7fba6c061a30 msgr2=0x7fba6c0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:38.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.487+0000 7fba71b01640 1 -- 172.21.15.7:0/515766973 shutdown_connections 2023-11-25T05:04:38.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.487+0000 7fba71b01640 1 -- 172.21.15.7:0/515766973 wait complete. 2023-11-25T05:04:38.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.487+0000 7fba71b01640 10 monclient: build_initial_monmap 2023-11-25T05:04:38.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.487+0000 7fba71b01640 10 monclient: monmap: 2023-11-25T05:04:38.494 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:38.494 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:38.494 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:38.494 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:38.494 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:38.494 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.494 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:38.494 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:38.494 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:38.494 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.488+0000 7fba71b01640 1 librados: starting msgr at 2023-11-25T05:04:38.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.488+0000 7fba71b01640 1 librados: starting objecter 2023-11-25T05:04:38.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.488+0000 7fba71b01640 1 Processor -- start 2023-11-25T05:04:38.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.488+0000 7fba71b01640 1 -- start start 2023-11-25T05:04:38.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.488+0000 7fba71b01640 1 librados: setting wanted keys 2023-11-25T05:04:38.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.488+0000 7fba71b01640 1 librados: calling monclient init 2023-11-25T05:04:38.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.488+0000 7fba71b01640 10 monclient: init 2023-11-25T05:04:38.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.488+0000 7fba71b01640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:38.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.488+0000 7fba71b01640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:04:38.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.488+0000 7fba71b01640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba6c0a8a30 0x7fba6c1daa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.488+0000 7fba71b01640 10 monclient(hunting): picked mon.noname-a con 0x7fba6c0a8a30 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:38.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.488+0000 7fba71b01640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fba6c0a8e10 0x7fba6c1daf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.488+0000 7fba71b01640 10 monclient(hunting): picked mon.noname-c con 0x7fba6c0a8e10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:38.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.488+0000 7fba71b01640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba6c142bf0 0x7fba6c0ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba71b01640 10 monclient(hunting): picked mon.noname-b con 0x7fba6c142bf0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:38.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba71b01640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba71b01640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c0ace70 con 0x7fba6c0a8a30 2023-11-25T05:04:38.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba71b01640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba71b01640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c1da530 con 0x7fba6c142bf0 2023-11-25T05:04:38.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba71b01640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba71b01640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c0ad8f0 con 0x7fba6c0a8e10 2023-11-25T05:04:38.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba71b01640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:38.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba71b01640 10 monclient(hunting): authenticate will time out at 1404.731891s 2023-11-25T05:04:38.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba6bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba6c142bf0 0x7fba6c0ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:38.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba6bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba6c142bf0 0x7fba6c0ae500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:35218/0 (socket says 172.21.15.7:35218) 2023-11-25T05:04:38.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba6bfff640 1 -- 172.21.15.7:0/1622091393 learned_addr learned my addr 172.21.15.7:0/1622091393 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:38.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba6bfff640 10 monclient(hunting): get_auth_request con 0x7fba6c142bf0 auth_method 0 2023-11-25T05:04:38.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:38.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba6bfff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:38.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.489+0000 7fba6bfff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:38.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.490+0000 7fba6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:38.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.490+0000 7fba6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:38.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.490+0000 7fba6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:38.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.493+0000 7fba6bfff640 10 monclient(hunting): handle_auth_done global_id 4357 payload 824 2023-11-25T05:04:38.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.493+0000 7fba6bfff640 1 -- 172.21.15.7:0/1622091393 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fba6c0a8e10 msgr2=0x7fba6c1daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.493+0000 7fba6bfff640 1 --2- 172.21.15.7:0/1622091393 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fba6c0a8e10 0x7fba6c1daf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.493+0000 7fba6bfff640 1 -- 172.21.15.7:0/1622091393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba6c0a8a30 msgr2=0x7fba6c1daa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.493+0000 7fba6bfff640 1 --2- 172.21.15.7:0/1622091393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba6c0a8a30 0x7fba6c1daa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.493+0000 7fba6bfff640 10 monclient: _finish_hunting 0 2023-11-25T05:04:38.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.493+0000 7fba6bfff640 1 monclient: found mon.noname-b 2023-11-25T05:04:38.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.493+0000 7fba6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:38.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.493+0000 7fba6bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:38.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.493+0000 7fba6bfff640 1 -- 172.21.15.7:0/1622091393 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c0094d0 con 0x7fba6c142bf0 2023-11-25T05:04:38.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.496+0000 7fba6bfff640 1 --2- 172.21.15.7:0/1622091393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba6c142bf0 0x7fba6c0ae500 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fba6400b970 tx=0x7fba6400be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:38.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.496+0000 7fba53fff640 10 client.?.objecter ms_handle_connect 0x7fba6c142bf0 2023-11-25T05:04:38.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.496+0000 7fba53fff640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:38.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba53fff640 1 -- 172.21.15.7:0/1622091393 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6400cb40 con 0x7fba6c142bf0 2023-11-25T05:04:38.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba53fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:38.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba53fff640 10 monclient: dump: 2023-11-25T05:04:38.502 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.502 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.502 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.502 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.502 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.502 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.502 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.502 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.502 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.502 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba53fff640 10 monclient: _finish_auth 0 2023-11-25T05:04:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba53fff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba53fff640 1 -- 172.21.15.7:0/1622091393 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6400ccc0 con 0x7fba6c142bf0 2023-11-25T05:04:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba71b01640 5 monclient: authenticate success, global_id 4357 2023-11-25T05:04:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba71b01640 10 monclient: _renew_subs 2023-11-25T05:04:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba71b01640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba71b01640 1 -- 172.21.15.7:0/1622091393 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba6c0adb70 con 0x7fba6c142bf0 2023-11-25T05:04:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba53fff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba53fff640 1 -- 172.21.15.7:0/1622091393 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba64014680 con 0x7fba6c142bf0 2023-11-25T05:04:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba53fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba53fff640 10 monclient: dump: 2023-11-25T05:04:38.504 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.504 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.504 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.504 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.504 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.504 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.504 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.504 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.504 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.504 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba71b01640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:38.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba71b01640 10 monclient: _renew_subs 2023-11-25T05:04:38.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba71b01640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:38.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba71b01640 1 -- 172.21.15.7:0/1622091393 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6c0b0600 con 0x7fba6c142bf0 2023-11-25T05:04:38.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba71b01640 1 librados: init done 2023-11-25T05:04:38.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.500+0000 7fba69ffb640 4 set_mon_vals no callback set 2023-11-25T05:04:38.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.501+0000 7fba71b01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:38.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.501+0000 7fba71b01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:38.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.501+0000 7fba71b01640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:38.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.501+0000 7fba71b01640 1 -- 172.21.15.7:0/1622091393 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6c067ac0 con 0x7fba6c142bf0 2023-11-25T05:04:38.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.560+0000 7f4c2bfff640 1 -- 172.21.15.7:0/1542095784 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910197985 0 0) 0x7f4c340a8cf0 con 0x7f4c34141f90 2023-11-25T05:04:38.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.560+0000 7f4c2bfff640 10 monclient(hunting): _negotiate 2023-11-25T05:04:38.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.560+0000 7f4c2bfff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:38.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.560+0000 7f4c2bfff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:38.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.560+0000 7f4c2bfff640 10 monclient(hunting): my global_id is 4326 2023-11-25T05:04:38.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.560+0000 7f4c2bfff640 1 -- 172.21.15.7:0/1542095784 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c20003520 con 0x7f4c34141f90 2023-11-25T05:04:38.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 1 -- 172.21.15.7:0/1542095784 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2089624327 0 0) 0x7f4c20003520 con 0x7f4c34141f90 2023-11-25T05:04:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 1 -- 172.21.15.7:0/1542095784 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c340a8910 msgr2=0x7f4c3413ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 1 --2- 172.21.15.7:0/1542095784 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c340a8910 0x7f4c3413ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 1 -- 172.21.15.7:0/1542095784 >> v1:172.21.15.90:6789/0 conn(0x7f4c340a9b70 legacy=0x7f4c3413f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 10 monclient: _finish_hunting 0 2023-11-25T05:04:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 1 monclient: found mon.noname-a-legacy 2023-11-25T05:04:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.7:6789/0 2023-11-25T05:04:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 1 -- 172.21.15.7:0/1542095784 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c340a9720 con 0x7f4c34141f90 2023-11-25T05:04:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 10 monclient: _finish_auth 0 2023-11-25T05:04:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 1 -- 172.21.15.7:0/1542095784 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4243585761 0 0) 0x7f4c2c0019f0 con 0x7f4c34141f90 2023-11-25T05:04:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-11-25T05:04:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 10 monclient: dump: 2023-11-25T05:04:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.567 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] but i'm connected to v1:172.21.15.7:6789/0, reconnecting 2023-11-25T05:04:38.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:38.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.563+0000 7f4c2bfff640 1 -- 172.21.15.7:0/1542095784 >> v1:172.21.15.7:6789/0 conn(0x7f4c34141f90 legacy=0x7f4c341423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.564+0000 7f4c2bfff640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:04:38.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.564+0000 7f4c2bfff640 1 --2- 172.21.15.7:0/1542095784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c200039a0 0x7f4c20003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.564+0000 7f4c2bfff640 10 monclient(hunting): picked mon.a con 0x7f4c200039a0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:38.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.564+0000 7f4c2bfff640 1 --2- 172.21.15.7:0/1542095784 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c340a8910 0x7f4c200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.564+0000 7f4c2bfff640 10 monclient(hunting): picked mon.c con 0x7f4c340a8910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:38.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.564+0000 7f4c2bfff640 1 --2- 172.21.15.7:0/1542095784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c20004e20 0x7f4c2000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.564+0000 7f4c2bfff640 10 monclient(hunting): picked mon.b con 0x7f4c20004e20 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:38.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.564+0000 7f4c2bfff640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.564+0000 7f4c2bfff640 1 -- 172.21.15.7:0/1542095784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c340aaeb0 con 0x7f4c200039a0 2023-11-25T05:04:38.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.564+0000 7f4c2bfff640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.564+0000 7f4c2bfff640 1 -- 172.21.15.7:0/1542095784 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c200054a0 con 0x7f4c20004e20 2023-11-25T05:04:38.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.564+0000 7f4c2bfff640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.564+0000 7f4c2bfff640 1 -- 172.21.15.7:0/1542095784 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c20005660 con 0x7f4c340a8910 2023-11-25T05:04:38.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.564+0000 7f4c2bfff640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:38.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.564+0000 7f4c2bfff640 10 monclient(hunting): _finish_auth 0 2023-11-25T05:04:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.565+0000 7f4c39385640 1 --2- 172.21.15.7:0/1542095784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c200039a0 0x7f4c20003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.566+0000 7f4c39385640 10 monclient(hunting): get_auth_request con 0x7f4c200039a0 auth_method 0 2023-11-25T05:04:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.566+0000 7f4c39385640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.566+0000 7f4c39385640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.566+0000 7f4c39385640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:04:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.566+0000 7f4c39385640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.566+0000 7f4c39385640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.566+0000 7f4c39385640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-11-25T05:04:38.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.568+0000 7f4c39385640 10 monclient(hunting): handle_auth_done global_id 4326 payload 293 2023-11-25T05:04:38.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.568+0000 7f4c39385640 1 -- 172.21.15.7:0/1542095784 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c340a8910 msgr2=0x7f4c200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:38.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.568+0000 7f4c39385640 1 --2- 172.21.15.7:0/1542095784 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c340a8910 0x7f4c200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.568+0000 7f4c39385640 1 -- 172.21.15.7:0/1542095784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c20004e20 msgr2=0x7f4c2000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:38.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.568+0000 7f4c39385640 1 --2- 172.21.15.7:0/1542095784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c20004e20 0x7f4c2000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.568+0000 7f4c39385640 10 monclient: _finish_hunting 0 2023-11-25T05:04:38.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.568+0000 7f4c39385640 1 monclient: found mon.a 2023-11-25T05:04:38.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.568+0000 7f4c39385640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:38.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.568+0000 7f4c39385640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:38.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.568+0000 7f4c39385640 1 -- 172.21.15.7:0/1542095784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c340a9720 con 0x7f4c200039a0 2023-11-25T05:04:38.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.568+0000 7f4c39385640 10 monclient: _finish_auth 0 2023-11-25T05:04:38.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.568+0000 7f4c39385640 10 monclient: _check_auth_tickets 2023-11-25T05:04:38.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.568+0000 7f4c39385640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:38.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.568+0000 7f4c3b5e9640 5 monclient: authenticate success, global_id 4326 2023-11-25T05:04:38.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.568+0000 7f4c3b5e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:38.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.570+0000 7f4c39385640 1 --2- 172.21.15.7:0/1542095784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c200039a0 0x7f4c20003dc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4c30009860 tx=0x7f4c200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3008303a1494ae server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:38.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.575+0000 7f4c2bfff640 1 -- 172.21.15.7:0/1542095784 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c30032600 con 0x7f4c200039a0 2023-11-25T05:04:38.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.575+0000 7f4c2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.575+0000 7f4c2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:38.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.575+0000 7f4c2bfff640 10 monclient: dump: 2023-11-25T05:04:38.577 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.577 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.577 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.577 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.577 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.577 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.577 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.577 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.578 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.578 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.575+0000 7f4c3b5e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:38.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.575+0000 7f4c2bfff640 1 -- 172.21.15.7:0/1542095784 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3002e050 con 0x7f4c200039a0 2023-11-25T05:04:38.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.575+0000 7f4c2bfff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:38.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.575+0000 7f4c2bfff640 1 -- 172.21.15.7:0/1542095784 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c300329c0 con 0x7f4c200039a0 2023-11-25T05:04:38.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.575+0000 7f4c3b5e9640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:38.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.575+0000 7f4c3b5e9640 4 set_mon_vals no callback set 2023-11-25T05:04:38.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.575+0000 7f4c2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.575+0000 7f4c2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:38.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.575+0000 7f4c2bfff640 10 monclient: dump: 2023-11-25T05:04:38.580 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.580 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.581 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.581 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.581 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.581 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.581 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.581 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.581 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.581 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.576+0000 7f4c3b5e9640 10 monclient: shutdown 2023-11-25T05:04:38.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.576+0000 7f4c3b5e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:38.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.576+0000 7f4c3b5e9640 1 -- 172.21.15.7:0/1542095784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c200039a0 msgr2=0x7f4c20003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.576+0000 7f4c3b5e9640 1 --2- 172.21.15.7:0/1542095784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c200039a0 0x7f4c20003dc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4c30009860 tx=0x7f4c200057f0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.576+0000 7f4c39b86640 1 -- 172.21.15.7:0/1542095784 reap_dead start 2023-11-25T05:04:38.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.576+0000 7f4c3b5e9640 1 -- 172.21.15.7:0/1542095784 shutdown_connections 2023-11-25T05:04:38.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.576+0000 7f4c3b5e9640 1 -- 172.21.15.7:0/1542095784 >> 172.21.15.7:0/1542095784 conn(0x7f4c340b7780 msgr2=0x7f4c340b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:38.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.576+0000 7f4c3b5e9640 1 -- 172.21.15.7:0/1542095784 shutdown_connections 2023-11-25T05:04:38.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.576+0000 7f4c3b5e9640 1 -- 172.21.15.7:0/1542095784 wait complete. 2023-11-25T05:04:38.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.577+0000 7f4c3b5e9640 10 monclient: build_initial_monmap 2023-11-25T05:04:38.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.577+0000 7f4c3b5e9640 10 monclient: monmap: 2023-11-25T05:04:38.586 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:38.586 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:38.586 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:38.586 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:38.586 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:38.586 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.586 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:38.586 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:38.586 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:38.586 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.577+0000 7f4c3b5e9640 1 librados: starting msgr at 2023-11-25T05:04:38.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.577+0000 7f4c3b5e9640 1 librados: starting objecter 2023-11-25T05:04:38.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.577+0000 7f4c3b5e9640 1 Processor -- start 2023-11-25T05:04:38.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.577+0000 7f4c3b5e9640 1 -- start start 2023-11-25T05:04:38.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 1 librados: setting wanted keys 2023-11-25T05:04:38.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 1 librados: calling monclient init 2023-11-25T05:04:38.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 10 monclient: init 2023-11-25T05:04:38.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:38.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:04:38.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c200039a0 0x7f4c341d6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 10 monclient(hunting): picked mon.noname-a con 0x7f4c200039a0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:38.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c341378a0 0x7f4c34137ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 10 monclient(hunting): picked mon.noname-c con 0x7f4c341378a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:38.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c3406e2e0 0x7f4c3406e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 10 monclient(hunting): picked mon.noname-b con 0x7f4c3406e2e0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:38.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c340ac0a0 con 0x7f4c200039a0 2023-11-25T05:04:38.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3406ddc0 con 0x7f4c3406e2e0 2023-11-25T05:04:38.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:38.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3406fa50 con 0x7f4c341378a0 2023-11-25T05:04:38.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:38.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.578+0000 7f4c3b5e9640 10 monclient(hunting): authenticate will time out at 1404.821778s 2023-11-25T05:04:38.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.579+0000 7f4c39385640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c200039a0 0x7f4c341d6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:38.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.580+0000 7f4c39385640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c200039a0 0x7f4c341d6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58394/0 (socket says 172.21.15.7:58394) 2023-11-25T05:04:38.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.580+0000 7f4c39385640 1 -- 172.21.15.7:0/2558276724 learned_addr learned my addr 172.21.15.7:0/2558276724 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:38.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.580+0000 7f4c39385640 10 monclient(hunting): get_auth_request con 0x7f4c200039a0 auth_method 0 2023-11-25T05:04:38.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.580+0000 7f4c39385640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:38.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.580+0000 7f4c39385640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:38.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.580+0000 7f4c39385640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:38.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.580+0000 7f4c39385640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:38.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.580+0000 7f4c39385640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:38.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.580+0000 7f4c39385640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:38.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.583+0000 7f4c39385640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2023-11-25T05:04:38.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.583+0000 7f4c39385640 1 -- 172.21.15.7:0/2558276724 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c341378a0 msgr2=0x7f4c34137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.583+0000 7f4c39385640 1 --2- 172.21.15.7:0/2558276724 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c341378a0 0x7f4c34137ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.583+0000 7f4c39385640 1 -- 172.21.15.7:0/2558276724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c3406e2e0 msgr2=0x7f4c3406e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.583+0000 7f4c39385640 1 --2- 172.21.15.7:0/2558276724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c3406e2e0 0x7f4c3406e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.583+0000 7f4c39385640 10 monclient: _finish_hunting 0 2023-11-25T05:04:38.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.583+0000 7f4c39385640 1 monclient: found mon.noname-a 2023-11-25T05:04:38.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.583+0000 7f4c39385640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:38.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.583+0000 7f4c39385640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:38.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.583+0000 7f4c39385640 1 -- 172.21.15.7:0/2558276724 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c34138220 con 0x7f4c200039a0 2023-11-25T05:04:38.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.586+0000 7f4c39385640 1 --2- 172.21.15.7:0/2558276724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c200039a0 0x7f4c341d6690 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4c30000c00 tx=0x7f4c300047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:38.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.586+0000 7f4c29ffb640 10 client.?.objecter ms_handle_connect 0x7f4c200039a0 2023-11-25T05:04:38.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.586+0000 7f4c29ffb640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:38.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c29ffb640 1 -- 172.21.15.7:0/2558276724 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3003d890 con 0x7f4c200039a0 2023-11-25T05:04:38.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c29ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:38.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c29ffb640 10 monclient: dump: 2023-11-25T05:04:38.594 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.594 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.594 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.594 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.594 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.594 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.594 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.594 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.594 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.594 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c29ffb640 10 monclient: _finish_auth 0 2023-11-25T05:04:38.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c29ffb640 10 monclient: _check_auth_tickets 2023-11-25T05:04:38.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:38.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c29ffb640 1 -- 172.21.15.7:0/2558276724 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3003d9f0 con 0x7f4c200039a0 2023-11-25T05:04:38.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c29ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:38.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c29ffb640 1 -- 172.21.15.7:0/2558276724 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3003dd80 con 0x7f4c200039a0 2023-11-25T05:04:38.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c3b5e9640 5 monclient: authenticate success, global_id 4329 2023-11-25T05:04:38.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c3b5e9640 10 monclient: _renew_subs 2023-11-25T05:04:38.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c3b5e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:38.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c3b5e9640 1 -- 172.21.15.7:0/2558276724 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c341d70b0 con 0x7f4c200039a0 2023-11-25T05:04:38.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c2b7fe640 4 set_mon_vals no callback set 2023-11-25T05:04:38.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:38.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c29ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:38.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c29ffb640 10 monclient: dump: 2023-11-25T05:04:38.596 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:38.597 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:38.597 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.597 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:38.597 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:38.597 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:38.597 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:38.597 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:38.597 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:38.598 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c3b5e9640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c3b5e9640 10 monclient: _renew_subs 2023-11-25T05:04:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c3b5e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c3b5e9640 1 -- 172.21.15.7:0/2558276724 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c3406ee60 con 0x7f4c200039a0 2023-11-25T05:04:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.590+0000 7f4c3b5e9640 1 librados: init done 2023-11-25T05:04:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.591+0000 7f4c3b5e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.591+0000 7f4c3b5e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.591+0000 7f4c3b5e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:38.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.591+0000 7f4c3b5e9640 1 -- 172.21.15.7:0/2558276724 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c3406df30 con 0x7f4c200039a0 2023-11-25T05:04:38.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.606+0000 7ffac57fa640 1 -- 172.21.15.7:0/420940907 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffac0054bb0 con 0x7ffac806d0b0 2023-11-25T05:04:38.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.606+0000 7ffac57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:38.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.606+0000 7ffac57fa640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:38.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.614+0000 7f4c29ffb640 1 -- 172.21.15.7:0/2558276724 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4c30039040 con 0x7f4c200039a0 2023-11-25T05:04:38.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.615+0000 7f4c29ffb640 1 --2- 172.21.15.7:0/2558276724 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f4c1806c850 0x7f4c1806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.615+0000 7f4c38b84640 1 --2- 172.21.15.7:0/2558276724 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f4c1806c850 0x7f4c1806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:38.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.615+0000 7f4c38b84640 10 monclient: get_auth_request con 0x7f4c1806c850 auth_method 0 2023-11-25T05:04:38.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.615+0000 7f4c29ffb640 10 client.4329.objecter ms_handle_connect 0x7f4c1806c850 2023-11-25T05:04:38.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.615+0000 7f4c38b84640 1 --2- 172.21.15.7:0/2558276724 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f4c1806c850 0x7f4c1806ed10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4c340aba10 tx=0x7f4c24007a40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:38.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.618+0000 7f4c29ffb640 1 -- 172.21.15.7:0/2558276724 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4c3007a510 con 0x7f4c200039a0 2023-11-25T05:04:38.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.618+0000 7f4c29ffb640 10 client.4329.objecter ms_dispatch 0x7f4c34000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:38.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.618+0000 7f4c29ffb640 3 client.4329.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:38.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.618+0000 7f4c29ffb640 3 client.4329.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:38.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.618+0000 7f4c29ffb640 20 client.4329.objecter dump_active .. 0 homeless 2023-11-25T05:04:38.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.709+0000 7f6b53abb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-11-25T05:04:38.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.709+0000 7f6b53abb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-11-25T05:04:38.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.709+0000 7f6b53abb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:38.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.709+0000 7f6b53abb640 1 -- 172.21.15.7:0/2769942322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6b4c075b60 con 0x7f6b4c1454a0 2023-11-25T05:04:38.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.741+0000 7fba53fff640 1 -- 172.21.15.7:0/1622091393 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fba6401d3b0 con 0x7fba6c142bf0 2023-11-25T05:04:38.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.742+0000 7fba53fff640 1 --2- 172.21.15.7:0/1622091393 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fba5806c7d0 0x7fba5806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:38.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.742+0000 7fba6b7fe640 1 --2- 172.21.15.7:0/1622091393 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fba5806c7d0 0x7fba5806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:38.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.742+0000 7fba6b7fe640 10 monclient: get_auth_request con 0x7fba5806c7d0 auth_method 0 2023-11-25T05:04:38.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.742+0000 7fba53fff640 10 client.4357.objecter ms_handle_connect 0x7fba5806c7d0 2023-11-25T05:04:38.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.742+0000 7fba6b7fe640 1 --2- 172.21.15.7:0/1622091393 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fba5806c7d0 0x7fba5806ec90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fba60001560 tx=0x7fba60007990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:38.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.744+0000 7fba53fff640 1 -- 172.21.15.7:0/1622091393 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fba6400ce30 con 0x7fba6c142bf0 2023-11-25T05:04:38.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.744+0000 7fba53fff640 10 client.4357.objecter ms_dispatch 0x7fba6c000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:38.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.744+0000 7fba53fff640 3 client.4357.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:38.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.744+0000 7fba53fff640 3 client.4357.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:38.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.745+0000 7fba53fff640 20 client.4357.objecter dump_active .. 0 homeless 2023-11-25T05:04:38.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.754+0000 7fba53fff640 1 -- 172.21.15.7:0/1622091393 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba64058580 con 0x7fba6c142bf0 2023-11-25T05:04:38.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.754+0000 7fba53fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:38.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.754+0000 7fba53fff640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:38.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.761+0000 7f6b427fc640 1 -- 172.21.15.7:0/2769942322 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6b44017090 con 0x7f6b4c1454a0 2023-11-25T05:04:38.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.761+0000 7f6b427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-11-25T05:04:38.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.761+0000 7f6b427fc640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:38.763 INFO:teuthology.orchestra.run.smithi007.stdout:55834574851 2023-11-25T05:04:38.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 10 librados: watch_flush enter 2023-11-25T05:04:38.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 10 librados: watch_flush exit 2023-11-25T05:04:38.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 20 client.4351.objecter shutdown clearing up homeless session... 2023-11-25T05:04:38.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 10 client.4351.objecter successfully canceled tick 2023-11-25T05:04:38.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 1 -- 172.21.15.7:0/2769942322 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6b3006c820 msgr2=0x7f6b3006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 1 --2- 172.21.15.7:0/2769942322 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6b3006c820 0x7f6b3006ece0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009710 tx=0x7f6b48006c90 comp rx=0 tx=0).stop 2023-11-25T05:04:38.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 10 monclient: shutdown 2023-11-25T05:04:38.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:38.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 1 -- 172.21.15.7:0/2769942322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b4c1454a0 msgr2=0x7f6b4c1df350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 1 --2- 172.21.15.7:0/2769942322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b4c1454a0 0x7f6b4c1df350 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400ca80 tx=0x7f6b4400cf40 comp rx=0 tx=0).stop 2023-11-25T05:04:38.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 1 -- 172.21.15.7:0/2769942322 shutdown_connections 2023-11-25T05:04:38.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 1 --2- 172.21.15.7:0/2769942322 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6b3006c820 0x7f6b3006ece0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 1 --2- 172.21.15.7:0/2769942322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b4c1454a0 0x7f6b4c1df350 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 1 --2- 172.21.15.7:0/2769942322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b4c0a8e10 0x7f6b4c1debd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 1 --2- 172.21.15.7:0/2769942322 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b4c0a8a30 0x7f6b4c13ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 1 -- 172.21.15.7:0/2769942322 >> 172.21.15.7:0/2769942322 conn(0x7f6b4c061a30 msgr2=0x7f6b4c1458f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:38.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 1 -- 172.21.15.7:0/2769942322 shutdown_connections 2023-11-25T05:04:38.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 1 -- 172.21.15.7:0/2769942322 wait complete. 2023-11-25T05:04:38.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.763+0000 7f6b53abb640 1 librados: shutdown 2023-11-25T05:04:38.776 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.5 2023-11-25T05:04:38.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.794+0000 7ffacf1a9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-11-25T05:04:38.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.794+0000 7ffacf1a9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-11-25T05:04:38.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.794+0000 7ffacf1a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:38.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.794+0000 7ffacf1a9640 1 -- 172.21.15.7:0/420940907 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ffac813be60 con 0x7ffac806d0b0 2023-11-25T05:04:38.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.800+0000 7ffac57fa640 1 -- 172.21.15.7:0/420940907 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffac813be60 con 0x7ffac806d0b0 2023-11-25T05:04:38.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.800+0000 7ffac57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-11-25T05:04:38.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.800+0000 7ffac57fa640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:38.802 INFO:teuthology.orchestra.run.smithi007.stdout:68719476740 2023-11-25T05:04:38.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.802+0000 7ffacf1a9640 10 librados: watch_flush enter 2023-11-25T05:04:38.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.802+0000 7ffacf1a9640 10 librados: watch_flush exit 2023-11-25T05:04:38.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 20 client.4298.objecter shutdown clearing up homeless session... 2023-11-25T05:04:38.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 10 client.4298.objecter successfully canceled tick 2023-11-25T05:04:38.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 1 -- 172.21.15.7:0/420940907 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ffaa806c820 msgr2=0x7ffaa806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 1 --2- 172.21.15.7:0/420940907 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ffaa806c820 0x7ffaa806ece0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffab80018b0 tx=0x7ffab8009040 comp rx=0 tx=0).stop 2023-11-25T05:04:38.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 10 monclient: shutdown 2023-11-25T05:04:38.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:38.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 1 -- 172.21.15.7:0/420940907 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffac806d0b0 msgr2=0x7ffac8139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 1 --2- 172.21.15.7:0/420940907 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffac806d0b0 0x7ffac8139720 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffac000be20 tx=0x7ffac000bf10 comp rx=0 tx=0).stop 2023-11-25T05:04:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 1 -- 172.21.15.7:0/420940907 shutdown_connections 2023-11-25T05:04:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 1 --2- 172.21.15.7:0/420940907 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ffaa806c820 0x7ffaa806ece0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 1 --2- 172.21.15.7:0/420940907 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffac806d0b0 0x7ffac8139720 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 1 --2- 172.21.15.7:0/420940907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffac806ccd0 0x7ffac8137d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 1 --2- 172.21.15.7:0/420940907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffac806c0e0 0x7ffac813b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 1 -- 172.21.15.7:0/420940907 >> 172.21.15.7:0/420940907 conn(0x7ffac80b5ca0 msgr2=0x7ffac8073c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 1 -- 172.21.15.7:0/420940907 shutdown_connections 2023-11-25T05:04:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 1 -- 172.21.15.7:0/420940907 wait complete. 2023-11-25T05:04:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.803+0000 7ffacf1a9640 1 librados: shutdown 2023-11-25T05:04:38.816 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476740 for osd.1 2023-11-25T05:04:38.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.881+0000 7f4c29ffb640 1 -- 172.21.15.7:0/2558276724 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c30076850 con 0x7f4c200039a0 2023-11-25T05:04:38.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.881+0000 7f4c29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:38.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.881+0000 7f4c29ffb640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:38.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-25T05:04:38.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.935+0000 7fba71b01640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-11-25T05:04:38.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.935+0000 7fba71b01640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-11-25T05:04:38.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.935+0000 7fba71b01640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:38.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.935+0000 7fba71b01640 1 -- 172.21.15.7:0/1622091393 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fba6c1da750 con 0x7fba6c142bf0 2023-11-25T05:04:38.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.938+0000 7fba53fff640 1 -- 172.21.15.7:0/1622091393 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fba64019070 con 0x7fba6c142bf0 2023-11-25T05:04:38.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.938+0000 7fba53fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-11-25T05:04:38.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.938+0000 7fba53fff640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:38.940 INFO:teuthology.orchestra.run.smithi007.stdout:68719476740 2023-11-25T05:04:38.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 10 librados: watch_flush enter 2023-11-25T05:04:38.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 10 librados: watch_flush exit 2023-11-25T05:04:38.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 20 client.4357.objecter shutdown clearing up homeless session... 2023-11-25T05:04:38.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 10 client.4357.objecter successfully canceled tick 2023-11-25T05:04:38.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 1 -- 172.21.15.7:0/1622091393 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fba5806c7d0 msgr2=0x7fba5806ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 1 --2- 172.21.15.7:0/1622091393 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fba5806c7d0 0x7fba5806ec90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fba60001560 tx=0x7fba60007990 comp rx=0 tx=0).stop 2023-11-25T05:04:38.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 10 monclient: shutdown 2023-11-25T05:04:38.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:38.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 1 -- 172.21.15.7:0/1622091393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba6c142bf0 msgr2=0x7fba6c0ae500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:38.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 1 --2- 172.21.15.7:0/1622091393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba6c142bf0 0x7fba6c0ae500 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fba6400b970 tx=0x7fba6400be30 comp rx=0 tx=0).stop 2023-11-25T05:04:38.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 1 -- 172.21.15.7:0/1622091393 shutdown_connections 2023-11-25T05:04:38.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 1 --2- 172.21.15.7:0/1622091393 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fba5806c7d0 0x7fba5806ec90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 1 --2- 172.21.15.7:0/1622091393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba6c142bf0 0x7fba6c0ae500 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 1 --2- 172.21.15.7:0/1622091393 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fba6c0a8e10 0x7fba6c1daf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 1 --2- 172.21.15.7:0/1622091393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba6c0a8a30 0x7fba6c1daa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:38.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 1 -- 172.21.15.7:0/1622091393 >> 172.21.15.7:0/1622091393 conn(0x7fba6c061a30 msgr2=0x7fba6c13cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:38.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 1 -- 172.21.15.7:0/1622091393 shutdown_connections 2023-11-25T05:04:38.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.940+0000 7fba71b01640 1 -- 172.21.15.7:0/1622091393 wait complete. 2023-11-25T05:04:38.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:38.941+0000 7fba71b01640 1 librados: shutdown 2023-11-25T05:04:38.954 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.3 2023-11-25T05:04:38.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-25T05:04:39.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.006+0000 7f8e17541640 10 monclient: get_monmap_and_config 2023-11-25T05:04:39.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.006+0000 7f8e17541640 10 monclient: build_initial_monmap 2023-11-25T05:04:39.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.006+0000 7f8e17541640 10 monclient: monmap: 2023-11-25T05:04:39.014 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:39.014 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.014 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:39.007420+0000 2023-11-25T05:04:39.014 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:39.007420+0000 2023-11-25T05:04:39.014 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:39.014 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.014 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:39.014 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:39.014 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:39.014 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:39.014 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:39.014 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.007+0000 7f8e17541640 1 Processor -- start 2023-11-25T05:04:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.007+0000 7f8e17541640 1 -- start start 2023-11-25T05:04:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.007+0000 7f8e17541640 10 monclient: init 2023-11-25T05:04:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.007+0000 7f8e17541640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.007+0000 7f8e17541640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-11-25T05:04:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.007+0000 7f8e17541640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e100a8a30 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.007+0000 7f8e17541640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e10142bf0 0x7f8e1013ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.007+0000 7f8e17541640 10 monclient(hunting): picked mon.noname-a con 0x7f8e10142bf0 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.007+0000 7f8e17541640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8e100a8e10 0x7f8e1013f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.007+0000 7f8e17541640 10 monclient(hunting): picked mon.noname-c con 0x7f8e100a8e10 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.007+0000 7f8e17541640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e100aac10 con 0x7f8e100a8a30 2023-11-25T05:04:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.007+0000 7f8e17541640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.007+0000 7f8e17541640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e100aaa10 con 0x7f8e10142bf0 2023-11-25T05:04:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.007+0000 7f8e17541640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.007+0000 7f8e17541640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e100aaef0 con 0x7f8e100a8e10 2023-11-25T05:04:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.008+0000 7f8e17541640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.008+0000 7f8e17541640 10 monclient(hunting): authenticate will time out at 1405.250838s 2023-11-25T05:04:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.008+0000 7f8e15ade640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8e100a8e10 0x7f8e1013f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.008+0000 7f8e14adc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e10142bf0 0x7f8e1013ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.008+0000 7f8e15ade640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8e100a8e10 0x7f8e1013f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:35238/0 (socket says 172.21.15.7:35238) 2023-11-25T05:04:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.008+0000 7f8e15ade640 1 -- 172.21.15.7:0/1842061344 learned_addr learned my addr 172.21.15.7:0/1842061344 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.008+0000 7f8e15ade640 10 monclient(hunting): get_auth_request con 0x7f8e100a8e10 auth_method 0 2023-11-25T05:04:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.008+0000 7f8e15ade640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.008+0000 7f8e15ade640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.008+0000 7f8e15ade640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.009+0000 7f8e14adc640 10 monclient(hunting): get_auth_request con 0x7f8e10142bf0 auth_method 0 2023-11-25T05:04:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.009+0000 7f8e14adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.009+0000 7f8e14adc640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.009+0000 7f8e14adc640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.009+0000 7f8e15ade640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.009+0000 7f8e15ade640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.009+0000 7f8e15ade640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.009+0000 7f8e14adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.009+0000 7f8e14adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.009+0000 7f8e14adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.011+0000 7f8e15ade640 10 monclient(hunting): handle_auth_done global_id 4363 payload 274 2023-11-25T05:04:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.011+0000 7f8e15ade640 1 -- 172.21.15.7:0/1842061344 >> v2:172.21.15.7:3300/0 conn(0x7f8e10142bf0 msgr2=0x7f8e1013ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:39.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.011+0000 7f8e15ade640 1 --2- 172.21.15.7:0/1842061344 >> v2:172.21.15.7:3300/0 conn(0x7f8e10142bf0 0x7f8e1013ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.011+0000 7f8e15ade640 1 -- 172.21.15.7:0/1842061344 >> v1:172.21.15.90:6789/0 conn(0x7f8e100a8a30 legacy=0x7f8e101424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:39.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.011+0000 7f8e15ade640 10 monclient: _finish_hunting 0 2023-11-25T05:04:39.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.011+0000 7f8e15ade640 1 monclient: found mon.noname-c 2023-11-25T05:04:39.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.011+0000 7f8e15ade640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:39.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.011+0000 7f8e15ade640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:39.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.011+0000 7f8e15ade640 1 -- 172.21.15.7:0/1842061344 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e100a97f0 con 0x7f8e100a8e10 2023-11-25T05:04:39.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.013+0000 7f8e15ade640 1 --2- 172.21.15.7:0/1842061344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e100a8e10 0x7f8e1013f230 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8e100b3a60 tx=0x7f8e0802bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab152382c54ac7e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e07fff640 1 -- 172.21.15.7:0/1842061344 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0802cb20 con 0x7f8e100a8e10 2023-11-25T05:04:39.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e07fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:39.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e07fff640 10 monclient: dump: 2023-11-25T05:04:39.019 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.019 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.020 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.020 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.020 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.020 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.020 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.020 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.020 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.020 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e07fff640 10 monclient: _finish_auth 0 2023-11-25T05:04:39.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e07fff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:39.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:39.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e17541640 5 monclient: authenticate success, global_id 4363 2023-11-25T05:04:39.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e17541640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:39.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e07fff640 1 -- 172.21.15.7:0/1842061344 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0802cc80 con 0x7f8e100a8e10 2023-11-25T05:04:39.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e07fff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:39.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e17541640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:39.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e17541640 4 set_mon_vals no callback set 2023-11-25T05:04:39.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e07fff640 1 -- 172.21.15.7:0/1842061344 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e080343f0 con 0x7f8e100a8e10 2023-11-25T05:04:39.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e07fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.017+0000 7f8e07fff640 10 monclient: dump: 2023-11-25T05:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.018+0000 7f8e17541640 10 monclient: shutdown 2023-11-25T05:04:39.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.018+0000 7f8e17541640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.018+0000 7f8e17541640 1 -- 172.21.15.7:0/1842061344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e100a8e10 msgr2=0x7f8e1013f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.018+0000 7f8e17541640 1 --2- 172.21.15.7:0/1842061344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e100a8e10 0x7f8e1013f230 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8e100b3a60 tx=0x7f8e0802bf80 comp rx=0 tx=0).stop 2023-11-25T05:04:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.018+0000 7f8e17541640 1 -- 172.21.15.7:0/1842061344 shutdown_connections 2023-11-25T05:04:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.018+0000 7f8e17541640 1 --2- 172.21.15.7:0/1842061344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e100a8e10 0x7f8e1013f230 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.018+0000 7f8e17541640 1 --2- 172.21.15.7:0/1842061344 >> v2:172.21.15.7:3300/0 conn(0x7f8e10142bf0 0x7f8e1013ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.018+0000 7f8e17541640 1 -- 172.21.15.7:0/1842061344 >> 172.21.15.7:0/1842061344 conn(0x7f8e10061a30 msgr2=0x7f8e100b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.018+0000 7f8e17541640 1 -- 172.21.15.7:0/1842061344 shutdown_connections 2023-11-25T05:04:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.018+0000 7f8e17541640 1 -- 172.21.15.7:0/1842061344 wait complete. 2023-11-25T05:04:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.019+0000 7f8e17541640 10 monclient: build_initial_monmap 2023-11-25T05:04:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.019+0000 7f8e17541640 10 monclient: monmap: 2023-11-25T05:04:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:39.030 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:39.030 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:39.030 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:39.030 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.030 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:39.030 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:39.030 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:39.030 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.019+0000 7f8e17541640 1 librados: starting msgr at 2023-11-25T05:04:39.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.019+0000 7f8e17541640 1 librados: starting objecter 2023-11-25T05:04:39.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.019+0000 7f8e17541640 1 Processor -- start 2023-11-25T05:04:39.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 1 -- start start 2023-11-25T05:04:39.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 1 librados: setting wanted keys 2023-11-25T05:04:39.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 1 librados: calling monclient init 2023-11-25T05:04:39.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 10 monclient: init 2023-11-25T05:04:39.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:39.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-25T05:04:39.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8e100a8a30 0x7f8e100ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 10 monclient(hunting): picked mon.noname-c con 0x7f8e100a8a30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:39.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e100a8e10 0x7f8e100b0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 10 monclient(hunting): picked mon.noname-a con 0x7f8e100a8e10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:39.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e10142bf0 0x7f8e100b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 10 monclient(hunting): picked mon.noname-b con 0x7f8e10142bf0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:39.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e100acc30 con 0x7f8e100a8e10 2023-11-25T05:04:39.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e100b0fd0 con 0x7f8e10142bf0 2023-11-25T05:04:39.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e100ab710 con 0x7f8e100a8a30 2023-11-25T05:04:39.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:39.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.020+0000 7f8e17541640 10 monclient(hunting): authenticate will time out at 1405.263494s 2023-11-25T05:04:39.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e15ade640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e10142bf0 0x7f8e100b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e152dd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8e100a8a30 0x7f8e100ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e14adc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e100a8e10 0x7f8e100b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e15ade640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e10142bf0 0x7f8e100b0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:35244/0 (socket says 172.21.15.7:35244) 2023-11-25T05:04:39.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e15ade640 1 -- 172.21.15.7:0/3918997230 learned_addr learned my addr 172.21.15.7:0/3918997230 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:39.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e15ade640 10 monclient(hunting): get_auth_request con 0x7f8e10142bf0 auth_method 0 2023-11-25T05:04:39.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e15ade640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e15ade640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e15ade640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e152dd640 10 monclient(hunting): get_auth_request con 0x7f8e100a8a30 auth_method 0 2023-11-25T05:04:39.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e152dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e152dd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e152dd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e14adc640 10 monclient(hunting): get_auth_request con 0x7f8e100a8e10 auth_method 0 2023-11-25T05:04:39.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e14adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e14adc640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.021+0000 7f8e14adc640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.022+0000 7f8e15ade640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.022+0000 7f8e15ade640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.022+0000 7f8e15ade640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.022+0000 7f8e152dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.022+0000 7f8e152dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.022+0000 7f8e152dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.022+0000 7f8e14adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.022+0000 7f8e14adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.022+0000 7f8e14adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.024+0000 7f8e15ade640 10 monclient(hunting): handle_auth_done global_id 4366 payload 824 2023-11-25T05:04:39.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.024+0000 7f8e15ade640 1 -- 172.21.15.7:0/3918997230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8e100a8a30 msgr2=0x7f8e100ac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.024+0000 7f8e15ade640 1 --2- 172.21.15.7:0/3918997230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8e100a8a30 0x7f8e100ac1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.024+0000 7f8e15ade640 1 -- 172.21.15.7:0/3918997230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e100a8e10 msgr2=0x7f8e100b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.024+0000 7f8e15ade640 1 --2- 172.21.15.7:0/3918997230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e100a8e10 0x7f8e100b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.024+0000 7f8e15ade640 10 monclient: _finish_hunting 0 2023-11-25T05:04:39.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.024+0000 7f8e15ade640 1 monclient: found mon.noname-b 2023-11-25T05:04:39.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.024+0000 7f8e15ade640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:39.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.024+0000 7f8e15ade640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:39.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.024+0000 7f8e15ade640 1 -- 172.21.15.7:0/3918997230 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e08009b30 con 0x7f8e10142bf0 2023-11-25T05:04:39.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.026+0000 7f8e15ade640 1 --2- 172.21.15.7:0/3918997230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e10142bf0 0x7f8e100b0a90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8e08009f40 tx=0x7f8e08004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.026+0000 7f8e05ffb640 10 client.?.objecter ms_handle_connect 0x7f8e10142bf0 2023-11-25T05:04:39.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.026+0000 7f8e05ffb640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:39.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e05ffb640 1 -- 172.21.15.7:0/3918997230 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0802c920 con 0x7f8e10142bf0 2023-11-25T05:04:39.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e05ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:39.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e05ffb640 10 monclient: dump: 2023-11-25T05:04:39.037 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.037 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.037 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.037 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.037 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.038 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.038 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.038 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.038 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.039 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e05ffb640 10 monclient: _finish_auth 0 2023-11-25T05:04:39.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e05ffb640 10 monclient: _check_auth_tickets 2023-11-25T05:04:39.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:39.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e05ffb640 1 -- 172.21.15.7:0/3918997230 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0802caa0 con 0x7f8e10142bf0 2023-11-25T05:04:39.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e05ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:39.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e05ffb640 1 -- 172.21.15.7:0/3918997230 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0803e690 con 0x7f8e10142bf0 2023-11-25T05:04:39.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e17541640 5 monclient: authenticate success, global_id 4366 2023-11-25T05:04:39.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e17541640 10 monclient: _renew_subs 2023-11-25T05:04:39.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e17541640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:39.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e17541640 1 -- 172.21.15.7:0/3918997230 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e100ac730 con 0x7f8e10142bf0 2023-11-25T05:04:39.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e077fe640 4 set_mon_vals no callback set 2023-11-25T05:04:39.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e05ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:39.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e05ffb640 10 monclient: dump: 2023-11-25T05:04:39.040 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.040 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.040 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.040 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.040 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.040 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.040 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.040 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.040 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.040 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e17541640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:39.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e17541640 10 monclient: _renew_subs 2023-11-25T05:04:39.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e17541640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:39.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e17541640 1 -- 172.21.15.7:0/3918997230 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e100abc10 con 0x7f8e10142bf0 2023-11-25T05:04:39.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.030+0000 7f8e17541640 1 librados: init done 2023-11-25T05:04:39.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.032+0000 7f8e17541640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:39.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.032+0000 7f8e17541640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:39.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.032+0000 7f8e17541640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:39.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.032+0000 7f8e17541640 1 -- 172.21.15.7:0/3918997230 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e100abdd0 con 0x7f8e10142bf0 2023-11-25T05:04:39.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.048+0000 7f8e05ffb640 1 -- 172.21.15.7:0/3918997230 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8e0803e7f0 con 0x7f8e10142bf0 2023-11-25T05:04:39.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.049+0000 7f8e05ffb640 1 --2- 172.21.15.7:0/3918997230 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f8df406c7d0 0x7f8df406ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.049+0000 7f8e152dd640 1 --2- 172.21.15.7:0/3918997230 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f8df406c7d0 0x7f8df406ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.049+0000 7f8e152dd640 10 monclient: get_auth_request con 0x7f8df406c7d0 auth_method 0 2023-11-25T05:04:39.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.050+0000 7f8e152dd640 1 --2- 172.21.15.7:0/3918997230 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f8df406c7d0 0x7f8df406ec90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c004f60 tx=0x7f8e0c007e30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.050+0000 7f8e05ffb640 10 client.4366.objecter ms_handle_connect 0x7f8df406c7d0 2023-11-25T05:04:39.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.052+0000 7f8e05ffb640 1 -- 172.21.15.7:0/3918997230 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8e0807b590 con 0x7f8e10142bf0 2023-11-25T05:04:39.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.052+0000 7f8e05ffb640 10 client.4366.objecter ms_dispatch 0x7f8e10000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:39.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.052+0000 7f8e05ffb640 3 client.4366.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:39.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.052+0000 7f8e05ffb640 3 client.4366.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:39.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.052+0000 7f8e05ffb640 20 client.4366.objecter dump_active .. 0 homeless 2023-11-25T05:04:39.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.071+0000 7f4c3b5e9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-11-25T05:04:39.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.071+0000 7f4c3b5e9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-11-25T05:04:39.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.071+0000 7f4c3b5e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:39.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.071+0000 7f4c3b5e9640 1 -- 172.21.15.7:0/2558276724 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4c34075b60 con 0x7f4c200039a0 2023-11-25T05:04:39.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.077+0000 7f4c29ffb640 1 -- 172.21.15.7:0/2558276724 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4c3004d090 con 0x7f4c200039a0 2023-11-25T05:04:39.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.077+0000 7f4c29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-11-25T05:04:39.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.077+0000 7f4c29ffb640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:39.082 INFO:teuthology.orchestra.run.smithi007.stdout:64424509446 2023-11-25T05:04:39.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 10 librados: watch_flush enter 2023-11-25T05:04:39.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 10 librados: watch_flush exit 2023-11-25T05:04:39.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 20 client.4329.objecter shutdown clearing up homeless session... 2023-11-25T05:04:39.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 10 client.4329.objecter successfully canceled tick 2023-11-25T05:04:39.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 1 -- 172.21.15.7:0/2558276724 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f4c1806c850 msgr2=0x7f4c1806ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 1 --2- 172.21.15.7:0/2558276724 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f4c1806c850 0x7f4c1806ed10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4c340aba10 tx=0x7f4c24007a40 comp rx=0 tx=0).stop 2023-11-25T05:04:39.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 10 monclient: shutdown 2023-11-25T05:04:39.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:39.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 1 -- 172.21.15.7:0/2558276724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c200039a0 msgr2=0x7f4c341d6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 1 --2- 172.21.15.7:0/2558276724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c200039a0 0x7f4c341d6690 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4c30000c00 tx=0x7f4c300047d0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 1 -- 172.21.15.7:0/2558276724 shutdown_connections 2023-11-25T05:04:39.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 1 --2- 172.21.15.7:0/2558276724 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f4c1806c850 0x7f4c1806ed10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 1 --2- 172.21.15.7:0/2558276724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c3406e2e0 0x7f4c3406e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 1 --2- 172.21.15.7:0/2558276724 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c341378a0 0x7f4c34137ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 1 --2- 172.21.15.7:0/2558276724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c200039a0 0x7f4c341d6690 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 1 -- 172.21.15.7:0/2558276724 >> 172.21.15.7:0/2558276724 conn(0x7f4c340b7780 msgr2=0x7f4c340b2080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:39.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 1 -- 172.21.15.7:0/2558276724 shutdown_connections 2023-11-25T05:04:39.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 1 -- 172.21.15.7:0/2558276724 wait complete. 2023-11-25T05:04:39.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.079+0000 7f4c3b5e9640 1 librados: shutdown 2023-11-25T05:04:39.092 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.4 2023-11-25T05:04:39.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.104+0000 7f633fe83640 10 monclient: get_monmap_and_config 2023-11-25T05:04:39.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.104+0000 7f633fe83640 10 monclient: build_initial_monmap 2023-11-25T05:04:39.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.104+0000 7f633fe83640 10 monclient: monmap: 2023-11-25T05:04:39.106 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:39.106 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.106 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:39.105580+0000 2023-11-25T05:04:39.106 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:39.105580+0000 2023-11-25T05:04:39.106 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:39.106 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.106 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:39.107 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:39.107 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:39.107 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:39.107 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:39.107 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.105+0000 7f633fe83640 1 Processor -- start 2023-11-25T05:04:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.105+0000 7f633fe83640 1 -- start start 2023-11-25T05:04:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.105+0000 7f633fe83640 10 monclient: init 2023-11-25T05:04:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.105+0000 7f633fe83640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633fe83640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-11-25T05:04:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633fe83640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63380a8790 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633fe83640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63380a9bf0 0x7f6338142a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633fe83640 10 monclient(hunting): picked mon.noname-b con 0x7f63380a9bf0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633fe83640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63380a8b70 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633fe83640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63380aacc0 con 0x7f63380a8b70 2023-11-25T05:04:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633fe83640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63380b5d20 con 0x7f63380a8790 2023-11-25T05:04:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633fe83640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633fe83640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63381424f0 con 0x7f63380a9bf0 2023-11-25T05:04:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633fe83640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633fe83640 10 monclient(hunting): authenticate will time out at 1405.348909s 2023-11-25T05:04:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633d41e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63380a9bf0 0x7f6338142a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633e420640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f63380a8b70 0x7f63380b5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56684/0 (socket says 172.21.15.7:56684) 2023-11-25T05:04:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633e420640 1 -- 172.21.15.7:0/3926836639 learned_addr learned my addr 172.21.15.7:0/3926836639 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633d41e640 10 monclient(hunting): get_auth_request con 0x7f63380a9bf0 auth_method 0 2023-11-25T05:04:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633d41e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633d41e640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.106+0000 7f633d41e640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.107+0000 7f633d41e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.107+0000 7f633d41e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.107+0000 7f633d41e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.109+0000 7f633d41e640 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2023-11-25T05:04:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.109+0000 7f633d41e640 1 -- 172.21.15.7:0/3926836639 >> v1:172.21.15.90:6789/0 conn(0x7f63380a8790 legacy=0x7f63380b6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.109+0000 7f633d41e640 1 -- 172.21.15.7:0/3926836639 >> v1:172.21.15.7:6789/0 conn(0x7f63380a8b70 legacy=0x7f63380b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.109+0000 7f633d41e640 10 monclient: _finish_hunting 0 2023-11-25T05:04:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.109+0000 7f633d41e640 1 monclient: found mon.noname-b 2023-11-25T05:04:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.109+0000 7f633d41e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.109+0000 7f633d41e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:04:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.109+0000 7f633d41e640 1 -- 172.21.15.7:0/3926836639 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63380a9000 con 0x7f63380a9bf0 2023-11-25T05:04:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.111+0000 7f633d41e640 1 --2- 172.21.15.7:0/3926836639 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63380a9bf0 0x7f6338142a10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f63280097d0 tx=0x7f632802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48f409e1f79ce00 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633cc1d640 1 -- 172.21.15.7:0/3926836639 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6328031890 con 0x7f63380a9bf0 2023-11-25T05:04:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633cc1d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633cc1d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633cc1d640 10 monclient: dump: 2023-11-25T05:04:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.117 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633cc1d640 10 monclient: _finish_auth 0 2023-11-25T05:04:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633cc1d640 10 monclient: _check_auth_tickets 2023-11-25T05:04:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633cc1d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633cc1d640 1 -- 172.21.15.7:0/3926836639 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63280319f0 con 0x7f63380a9bf0 2023-11-25T05:04:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633fe83640 5 monclient: authenticate success, global_id 4304 2023-11-25T05:04:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633fe83640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633cc1d640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633fe83640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633fe83640 4 set_mon_vals no callback set 2023-11-25T05:04:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633cc1d640 1 -- 172.21.15.7:0/3926836639 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6328031d80 con 0x7f63380a9bf0 2023-11-25T05:04:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633cc1d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633cc1d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.114+0000 7f633cc1d640 10 monclient: dump: 2023-11-25T05:04:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.119 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.119 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.119 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.115+0000 7f633fe83640 10 monclient: shutdown 2023-11-25T05:04:39.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.115+0000 7f633fe83640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:39.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.115+0000 7f633fe83640 1 -- 172.21.15.7:0/3926836639 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63380a9bf0 msgr2=0x7f6338142a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.115+0000 7f633fe83640 1 --2- 172.21.15.7:0/3926836639 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63380a9bf0 0x7f6338142a10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f63280097d0 tx=0x7f632802b930 comp rx=0 tx=0).stop 2023-11-25T05:04:39.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.115+0000 7f633fe83640 1 -- 172.21.15.7:0/3926836639 shutdown_connections 2023-11-25T05:04:39.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.115+0000 7f633fe83640 1 --2- 172.21.15.7:0/3926836639 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63380a9bf0 0x7f6338142a10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.115+0000 7f633fe83640 1 -- 172.21.15.7:0/3926836639 >> 172.21.15.7:0/3926836639 conn(0x7f63380b41f0 msgr2=0x7f63380b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:39.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.115+0000 7f633fe83640 1 -- 172.21.15.7:0/3926836639 shutdown_connections 2023-11-25T05:04:39.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.115+0000 7f633fe83640 1 -- 172.21.15.7:0/3926836639 wait complete. 2023-11-25T05:04:39.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.116+0000 7f633fe83640 10 monclient: build_initial_monmap 2023-11-25T05:04:39.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.116+0000 7f633fe83640 10 monclient: monmap: 2023-11-25T05:04:39.123 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:39.123 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:39.123 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:39.123 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:39.123 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:39.123 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.123 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:39.124 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:39.124 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:39.124 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.116+0000 7f633fe83640 1 librados: starting msgr at 2023-11-25T05:04:39.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.116+0000 7f633fe83640 1 librados: starting objecter 2023-11-25T05:04:39.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.116+0000 7f633fe83640 1 Processor -- start 2023-11-25T05:04:39.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.116+0000 7f633fe83640 1 -- start start 2023-11-25T05:04:39.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.116+0000 7f633fe83640 1 librados: setting wanted keys 2023-11-25T05:04:39.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.116+0000 7f633fe83640 1 librados: calling monclient init 2023-11-25T05:04:39.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.116+0000 7f633fe83640 10 monclient: init 2023-11-25T05:04:39.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:39.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-25T05:04:39.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63380a8790 0x7f633813bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 10 monclient(hunting): picked mon.noname-c con 0x7f63380a8790 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:39.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63380a8b70 0x7f6338139a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 10 monclient(hunting): picked mon.noname-a con 0x7f63380a8b70 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:39.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63380a9bf0 0x7f63381316e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 10 monclient(hunting): picked mon.noname-b con 0x7f63380a9bf0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:39.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633813a780 con 0x7f63380a8b70 2023-11-25T05:04:39.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633813b260 con 0x7f63380a9bf0 2023-11-25T05:04:39.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f633813c4d0 con 0x7f63380a8790 2023-11-25T05:04:39.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:39.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633fe83640 10 monclient(hunting): authenticate will time out at 1405.360305s 2023-11-25T05:04:39.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.117+0000 7f633dc1f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63380a8790 0x7f633813bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.118+0000 7f633dc1f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63380a8790 0x7f633813bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41888/0 (socket says 172.21.15.7:41888) 2023-11-25T05:04:39.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.118+0000 7f633dc1f640 1 -- 172.21.15.7:0/3236682717 learned_addr learned my addr 172.21.15.7:0/3236682717 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:39.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.118+0000 7f633dc1f640 10 monclient(hunting): get_auth_request con 0x7f63380a8790 auth_method 0 2023-11-25T05:04:39.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.118+0000 7f633dc1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.118+0000 7f633dc1f640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.118+0000 7f633dc1f640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.118+0000 7f633dc1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.118+0000 7f633dc1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.118+0000 7f633dc1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.120+0000 7f633d41e640 1 --2- 172.21.15.7:0/3236682717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63380a8b70 0x7f6338139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.120+0000 7f633d41e640 10 monclient(hunting): get_auth_request con 0x7f63380a8b70 auth_method 0 2023-11-25T05:04:39.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.120+0000 7f633d41e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.120+0000 7f633d41e640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.120+0000 7f633d41e640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.120+0000 7f633d41e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.120+0000 7f633d41e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.121+0000 7f633d41e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.121+0000 7f633dc1f640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2023-11-25T05:04:39.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.121+0000 7f633dc1f640 1 -- 172.21.15.7:0/3236682717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63380a9bf0 msgr2=0x7f63381316e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.121+0000 7f633dc1f640 1 --2- 172.21.15.7:0/3236682717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63380a9bf0 0x7f63381316e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.121+0000 7f633dc1f640 1 -- 172.21.15.7:0/3236682717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63380a8b70 msgr2=0x7f6338139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.121+0000 7f633dc1f640 1 --2- 172.21.15.7:0/3236682717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63380a8b70 0x7f6338139a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.121+0000 7f633dc1f640 10 monclient: _finish_hunting 0 2023-11-25T05:04:39.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.121+0000 7f633dc1f640 1 monclient: found mon.noname-c 2023-11-25T05:04:39.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.121+0000 7f633dc1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:39.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.121+0000 7f633dc1f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:04:39.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.121+0000 7f633dc1f640 1 -- 172.21.15.7:0/3236682717 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6328009400 con 0x7f63380a8790 2023-11-25T05:04:39.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.124+0000 7f633dc1f640 1 --2- 172.21.15.7:0/3236682717 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63380a8790 0x7f633813bf90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f633400c470 tx=0x7f633400c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.124+0000 7f632e7fc640 10 client.?.objecter ms_handle_connect 0x7f63380a8790 2023-11-25T05:04:39.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.124+0000 7f632e7fc640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:39.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f632e7fc640 1 -- 172.21.15.7:0/3236682717 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6334005480 con 0x7f63380a8790 2023-11-25T05:04:39.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f632e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f632e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:39.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f632e7fc640 10 monclient: dump: 2023-11-25T05:04:39.131 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.132 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f632e7fc640 10 monclient: _finish_auth 0 2023-11-25T05:04:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f632e7fc640 10 monclient: _check_auth_tickets 2023-11-25T05:04:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f632e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f632e7fc640 1 -- 172.21.15.7:0/3236682717 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63340035f0 con 0x7f63380a8790 2023-11-25T05:04:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f633fe83640 5 monclient: authenticate success, global_id 4307 2023-11-25T05:04:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f633fe83640 10 monclient: _renew_subs 2023-11-25T05:04:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f633fe83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f633fe83640 1 -- 172.21.15.7:0/3236682717 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f633813b7b0 con 0x7f63380a8790 2023-11-25T05:04:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f632e7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f632e7fc640 1 -- 172.21.15.7:0/3236682717 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63340039a0 con 0x7f63380a8790 2023-11-25T05:04:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f632e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f632e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f632e7fc640 10 monclient: dump: 2023-11-25T05:04:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.134 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f632ffff640 4 set_mon_vals no callback set 2023-11-25T05:04:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f633fe83640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f633fe83640 10 monclient: _renew_subs 2023-11-25T05:04:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f633fe83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f633fe83640 1 -- 172.21.15.7:0/3236682717 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6338139fd0 con 0x7f63380a8790 2023-11-25T05:04:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.128+0000 7f633fe83640 1 librados: init done 2023-11-25T05:04:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.129+0000 7f633fe83640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.129+0000 7f633fe83640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.129+0000 7f633fe83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.129+0000 7f633fe83640 1 -- 172.21.15.7:0/3236682717 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6338075b60 con 0x7f63380a8790 2023-11-25T05:04:39.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-25T05:04:39.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-25T05:04:39.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.283+0000 7f8e05ffb640 1 -- 172.21.15.7:0/3918997230 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e080778d0 con 0x7f8e10142bf0 2023-11-25T05:04:39.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.283+0000 7f8e05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:39.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.283+0000 7f8e05ffb640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.371+0000 7f11acc21640 10 monclient: get_monmap_and_config 2023-11-25T05:04:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.371+0000 7f11acc21640 10 monclient: build_initial_monmap 2023-11-25T05:04:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.371+0000 7f11acc21640 10 monclient: monmap: 2023-11-25T05:04:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:39.372633+0000 2023-11-25T05:04:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:39.372633+0000 2023-11-25T05:04:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:39.375 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:39.375 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:39.375 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:39.375 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:39.375 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.372+0000 7f11acc21640 1 Processor -- start 2023-11-25T05:04:39.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.372+0000 7f11acc21640 1 -- start start 2023-11-25T05:04:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.372+0000 7f11acc21640 10 monclient: init 2023-11-25T05:04:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-11-25T05:04:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11a80a8910 0x7f11a813ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 10 monclient(hunting): picked mon.noname-b con 0x7f11a80a8910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f11a80a9b70 0x7f11a813f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 10 monclient(hunting): picked mon.noname-a con 0x7f11a80a9b70 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f11a8141f90 0x7f11a8142400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 10 monclient(hunting): picked mon.noname-c con 0x7f11a8141f90 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a80a8cf0 con 0x7f11a80a9b70 2023-11-25T05:04:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a813f6e0 con 0x7f11a8141f90 2023-11-25T05:04:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a80aabd0 con 0x7f11a80a8910 2023-11-25T05:04:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.373+0000 7f11acc21640 10 monclient(hunting): authenticate will time out at 1405.616173s 2023-11-25T05:04:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.374+0000 7f11a6d9d640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f11a8141f90 0x7f11a8142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.374+0000 7f11a5d9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f11a80a9b70 0x7f11a813f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.374+0000 7f11a6d9d640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f11a8141f90 0x7f11a8142400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:35256/0 (socket says 172.21.15.7:35256) 2023-11-25T05:04:39.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.374+0000 7f11a6d9d640 1 -- 172.21.15.7:0/3327090815 learned_addr learned my addr 172.21.15.7:0/3327090815 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:39.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.374+0000 7f11a6d9d640 10 monclient(hunting): get_auth_request con 0x7f11a8141f90 auth_method 0 2023-11-25T05:04:39.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.374+0000 7f11a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.374+0000 7f11a6d9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.374+0000 7f11a6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.374+0000 7f11a5d9b640 10 monclient(hunting): get_auth_request con 0x7f11a80a9b70 auth_method 0 2023-11-25T05:04:39.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.374+0000 7f11a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.374+0000 7f11a5d9b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.374+0000 7f11a5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.374+0000 7f11a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.374+0000 7f11a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.374+0000 7f11a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.375+0000 7f11a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.375+0000 7f11a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.375+0000 7f11a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.376+0000 7f11a6d9d640 10 monclient(hunting): handle_auth_done global_id 4369 payload 274 2023-11-25T05:04:39.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.376+0000 7f11a6d9d640 1 -- 172.21.15.7:0/3327090815 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11a80a8910 msgr2=0x7f11a813ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:39.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.376+0000 7f11a6d9d640 1 --2- 172.21.15.7:0/3327090815 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11a80a8910 0x7f11a813ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.376+0000 7f11a6d9d640 1 -- 172.21.15.7:0/3327090815 >> v2:172.21.15.7:3300/0 conn(0x7f11a80a9b70 msgr2=0x7f11a813f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:39.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.376+0000 7f11a6d9d640 1 --2- 172.21.15.7:0/3327090815 >> v2:172.21.15.7:3300/0 conn(0x7f11a80a9b70 0x7f11a813f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.376+0000 7f11a6d9d640 10 monclient: _finish_hunting 0 2023-11-25T05:04:39.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.376+0000 7f11a6d9d640 1 monclient: found mon.noname-c 2023-11-25T05:04:39.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.376+0000 7f11a6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:39.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.376+0000 7f11a6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:39.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.376+0000 7f11a6d9d640 1 -- 172.21.15.7:0/3327090815 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a80aaeb0 con 0x7f11a8141f90 2023-11-25T05:04:39.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.378+0000 7f11a6d9d640 1 --2- 172.21.15.7:0/3327090815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a8141f90 0x7f11a8142400 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f11a80a92b0 tx=0x7f11a002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed42aacbb2af46d4 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11a559a640 1 -- 172.21.15.7:0/3327090815 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a002ca70 con 0x7f11a8141f90 2023-11-25T05:04:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11a559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11a559a640 10 monclient: dump: 2023-11-25T05:04:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.387 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11a559a640 10 monclient: _finish_auth 0 2023-11-25T05:04:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11a559a640 10 monclient: _check_auth_tickets 2023-11-25T05:04:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11a559a640 1 -- 172.21.15.7:0/3327090815 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a002cbd0 con 0x7f11a8141f90 2023-11-25T05:04:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11a559a640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11a559a640 1 -- 172.21.15.7:0/3327090815 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a00343f0 con 0x7f11a8141f90 2023-11-25T05:04:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11acc21640 5 monclient: authenticate success, global_id 4369 2023-11-25T05:04:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11acc21640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11acc21640 4 set_mon_vals no callback set 2023-11-25T05:04:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11a559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:39.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.381+0000 7f11a559a640 10 monclient: dump: 2023-11-25T05:04:39.388 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.388 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.388 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.388 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.388 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.388 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.388 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.389 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.389 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.389 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.382+0000 7f11acc21640 10 monclient: shutdown 2023-11-25T05:04:39.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.382+0000 7f11acc21640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:39.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.382+0000 7f11acc21640 1 -- 172.21.15.7:0/3327090815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a8141f90 msgr2=0x7f11a8142400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.382+0000 7f11acc21640 1 --2- 172.21.15.7:0/3327090815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a8141f90 0x7f11a8142400 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f11a80a92b0 tx=0x7f11a002bed0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.382+0000 7f11acc21640 1 -- 172.21.15.7:0/3327090815 shutdown_connections 2023-11-25T05:04:39.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.382+0000 7f11acc21640 1 --2- 172.21.15.7:0/3327090815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a8141f90 0x7f11a8142400 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.382+0000 7f11acc21640 1 --2- 172.21.15.7:0/3327090815 >> v2:172.21.15.7:3300/0 conn(0x7f11a80a9b70 0x7f11a813f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.382+0000 7f11acc21640 1 --2- 172.21.15.7:0/3327090815 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11a80a8910 0x7f11a813ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.383+0000 7f11acc21640 1 -- 172.21.15.7:0/3327090815 >> 172.21.15.7:0/3327090815 conn(0x7f11a80b7780 msgr2=0x7f11a80b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:39.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.383+0000 7f11acc21640 1 -- 172.21.15.7:0/3327090815 shutdown_connections 2023-11-25T05:04:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.383+0000 7f11acc21640 1 -- 172.21.15.7:0/3327090815 wait complete. 2023-11-25T05:04:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.384+0000 7f11acc21640 10 monclient: build_initial_monmap 2023-11-25T05:04:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.384+0000 7f11acc21640 10 monclient: monmap: 2023-11-25T05:04:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:39.391 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.384+0000 7f11acc21640 1 librados: starting msgr at 2023-11-25T05:04:39.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.384+0000 7f11acc21640 1 librados: starting objecter 2023-11-25T05:04:39.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.384+0000 7f11acc21640 1 Processor -- start 2023-11-25T05:04:39.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.384+0000 7f11acc21640 1 -- start start 2023-11-25T05:04:39.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.384+0000 7f11acc21640 1 librados: setting wanted keys 2023-11-25T05:04:39.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.384+0000 7f11acc21640 1 librados: calling monclient init 2023-11-25T05:04:39.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.384+0000 7f11acc21640 10 monclient: init 2023-11-25T05:04:39.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:39.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-25T05:04:39.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11a80a8910 0x7f11a81c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 10 monclient(hunting): picked mon.noname-a con 0x7f11a80a8910 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a8141f90 0x7f11a80ae470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 10 monclient(hunting): picked mon.noname-b con 0x7f11a8141f90 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11a813a6a0 0x7f11a80b1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 10 monclient(hunting): picked mon.noname-c con 0x7f11a813a6a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a81c8940 con 0x7f11a80a8910 2023-11-25T05:04:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a81c7f70 con 0x7f11a8141f90 2023-11-25T05:04:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a81c9b00 con 0x7f11a813a6a0 2023-11-25T05:04:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.385+0000 7f11acc21640 10 monclient(hunting): authenticate will time out at 1405.628383s 2023-11-25T05:04:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.386+0000 7f11a659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11a80a8910 0x7f11a81c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.386+0000 7f11a5d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a8141f90 0x7f11a80ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.386+0000 7f11a5d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a8141f90 0x7f11a80ae470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:35260/0 (socket says 172.21.15.7:35260) 2023-11-25T05:04:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.386+0000 7f11a5d9b640 1 -- 172.21.15.7:0/3568551283 learned_addr learned my addr 172.21.15.7:0/3568551283 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.386+0000 7f11a5d9b640 10 monclient(hunting): get_auth_request con 0x7f11a8141f90 auth_method 0 2023-11-25T05:04:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.386+0000 7f11a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.386+0000 7f11a5d9b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.386+0000 7f11a5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.386+0000 7f11a659c640 10 monclient(hunting): get_auth_request con 0x7f11a80a8910 auth_method 0 2023-11-25T05:04:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.386+0000 7f11a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.386+0000 7f11a659c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.386+0000 7f11a659c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.387+0000 7f11a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.387+0000 7f11a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.387+0000 7f11a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.387+0000 7f11a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.387+0000 7f11a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.387+0000 7f11a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.389+0000 7f11a5d9b640 10 monclient(hunting): handle_auth_done global_id 4372 payload 824 2023-11-25T05:04:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.389+0000 7f11a5d9b640 1 -- 172.21.15.7:0/3568551283 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11a813a6a0 msgr2=0x7f11a80b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.389+0000 7f11a5d9b640 1 --2- 172.21.15.7:0/3568551283 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11a813a6a0 0x7f11a80b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.389+0000 7f11a5d9b640 1 -- 172.21.15.7:0/3568551283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11a80a8910 msgr2=0x7f11a81c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.389+0000 7f11a5d9b640 1 --2- 172.21.15.7:0/3568551283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11a80a8910 0x7f11a81c8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.389+0000 7f11a5d9b640 10 monclient: _finish_hunting 0 2023-11-25T05:04:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.389+0000 7f11a5d9b640 1 monclient: found mon.noname-b 2023-11-25T05:04:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.389+0000 7f11a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.389+0000 7f11a5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.389+0000 7f11a5d9b640 1 -- 172.21.15.7:0/3568551283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a0009b30 con 0x7f11a8141f90 2023-11-25T05:04:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.391+0000 7f11a5d9b640 1 --2- 172.21.15.7:0/3568551283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a8141f90 0x7f11a80ae470 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f119c00e9e0 tx=0x7f119c00ede0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.391+0000 7f1196ffd640 10 client.?.objecter ms_handle_connect 0x7f11a8141f90 2023-11-25T05:04:39.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.391+0000 7f1196ffd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:39.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f1196ffd640 1 -- 172.21.15.7:0/3568551283 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c009670 con 0x7f11a8141f90 2023-11-25T05:04:39.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f1196ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f1196ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:39.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f1196ffd640 10 monclient: dump: 2023-11-25T05:04:39.403 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.403 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.403 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.404 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.404 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.404 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.404 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.404 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.404 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.404 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f1196ffd640 10 monclient: _finish_auth 0 2023-11-25T05:04:39.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f1196ffd640 10 monclient: _check_auth_tickets 2023-11-25T05:04:39.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f1196ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:39.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f1196ffd640 1 -- 172.21.15.7:0/3568551283 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119c00c8c0 con 0x7f11a8141f90 2023-11-25T05:04:39.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f1196ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:39.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f1196ffd640 1 -- 172.21.15.7:0/3568551283 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c00cc70 con 0x7f11a8141f90 2023-11-25T05:04:39.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f1196ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f1196ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:39.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f1196ffd640 10 monclient: dump: 2023-11-25T05:04:39.405 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.405 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.405 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.405 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.405 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.405 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.405 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.406 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.406 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.406 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f11acc21640 5 monclient: authenticate success, global_id 4372 2023-11-25T05:04:39.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f11acc21640 10 monclient: _renew_subs 2023-11-25T05:04:39.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f11acc21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:39.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f11acc21640 1 -- 172.21.15.7:0/3568551283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a80b0250 con 0x7f11a8141f90 2023-11-25T05:04:39.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f11acc21640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:39.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f11acc21640 10 monclient: _renew_subs 2023-11-25T05:04:39.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f11a4d99640 4 set_mon_vals no callback set 2023-11-25T05:04:39.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f11acc21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:39.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f11acc21640 1 -- 172.21.15.7:0/3568551283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a80b15d0 con 0x7f11a8141f90 2023-11-25T05:04:39.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.395+0000 7f11acc21640 1 librados: init done 2023-11-25T05:04:39.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.396+0000 7f11acc21640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:39.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.396+0000 7f11acc21640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:39.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.396+0000 7f11acc21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:39.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.396+0000 7f11acc21640 1 -- 172.21.15.7:0/3568551283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a8075b60 con 0x7f11a8141f90 2023-11-25T05:04:39.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.398+0000 7f4456a0e640 10 monclient: get_monmap_and_config 2023-11-25T05:04:39.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.398+0000 7f4456a0e640 10 monclient: build_initial_monmap 2023-11-25T05:04:39.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.398+0000 7f4456a0e640 10 monclient: monmap: 2023-11-25T05:04:39.410 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:39.410 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.410 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:39.398965+0000 2023-11-25T05:04:39.410 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:39.398965+0000 2023-11-25T05:04:39.410 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:39.410 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.410 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:39.410 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:39.410 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:39.411 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:39.411 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:39.411 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.398+0000 7f4456a0e640 1 Processor -- start 2023-11-25T05:04:39.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.398+0000 7f4456a0e640 1 -- start start 2023-11-25T05:04:39.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.398+0000 7f4456a0e640 10 monclient: init 2023-11-25T05:04:39.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.399+0000 7f4456a0e640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:39.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.399+0000 7f4456a0e640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-11-25T05:04:39.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.399+0000 7f4456a0e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44500a8a30 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:39.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.399+0000 7f4456a0e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4450142bf0 0x7f445013ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.399+0000 7f4456a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f4450142bf0 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:39.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.399+0000 7f4456a0e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44500a8e10 0x7f445013f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.399+0000 7f4456a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f44500a8e10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:39.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.399+0000 7f4456a0e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44500aac10 con 0x7f44500a8a30 2023-11-25T05:04:39.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.399+0000 7f4456a0e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.399+0000 7f4456a0e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44500aaa10 con 0x7f4450142bf0 2023-11-25T05:04:39.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.399+0000 7f4456a0e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.399+0000 7f4456a0e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44500aaef0 con 0x7f44500a8e10 2023-11-25T05:04:39.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.399+0000 7f4456a0e640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:39.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.399+0000 7f4456a0e640 10 monclient(hunting): authenticate will time out at 1405.642584s 2023-11-25T05:04:39.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.403+0000 7f632e7fc640 1 -- 172.21.15.7:0/3236682717 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f633401b3b0 con 0x7f63380a8790 2023-11-25T05:04:39.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.403+0000 7f632e7fc640 1 --2- 172.21.15.7:0/3236682717 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f631c06c820 0x7f631c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.404+0000 7f633d41e640 1 --2- 172.21.15.7:0/3236682717 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f631c06c820 0x7f631c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.404+0000 7f633d41e640 10 monclient: get_auth_request con 0x7f631c06c820 auth_method 0 2023-11-25T05:04:39.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.404+0000 7f633d41e640 1 --2- 172.21.15.7:0/3236682717 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f631c06c820 0x7f631c06ece0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6328036000 tx=0x7f632803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.404+0000 7f632e7fc640 10 client.4307.objecter ms_handle_connect 0x7f631c06c820 2023-11-25T05:04:39.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.413+0000 7f1196ffd640 1 -- 172.21.15.7:0/3568551283 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f119c013470 con 0x7f11a8141f90 2023-11-25T05:04:39.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.414+0000 7f1196ffd640 1 --2- 172.21.15.7:0/3568551283 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f118806c7d0 0x7f118806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.414+0000 7f11a659c640 1 --2- 172.21.15.7:0/3568551283 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f118806c7d0 0x7f118806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.414+0000 7f11a659c640 10 monclient: get_auth_request con 0x7f118806c7d0 auth_method 0 2023-11-25T05:04:39.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.414+0000 7f11a659c640 1 --2- 172.21.15.7:0/3568551283 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f118806c7d0 0x7f118806ec90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1198009990 tx=0x7f1198009110 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.414+0000 7f1196ffd640 10 client.4372.objecter ms_handle_connect 0x7f118806c7d0 2023-11-25T05:04:39.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.416+0000 7f444f7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4450142bf0 0x7f445013ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.417+0000 7f1196ffd640 1 -- 172.21.15.7:0/3568551283 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f119c059010 con 0x7f11a8141f90 2023-11-25T05:04:39.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.417+0000 7f1196ffd640 10 client.4372.objecter ms_dispatch 0x7f11a8000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:39.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.417+0000 7f1196ffd640 3 client.4372.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:39.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.417+0000 7f1196ffd640 3 client.4372.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:39.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.417+0000 7f1196ffd640 20 client.4372.objecter dump_active .. 0 homeless 2023-11-25T05:04:39.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.418+0000 7f444f7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4450142bf0 0x7f445013ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58474/0 (socket says 172.21.15.7:58474) 2023-11-25T05:04:39.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.418+0000 7f444f7fe640 1 -- 172.21.15.7:0/3301154262 learned_addr learned my addr 172.21.15.7:0/3301154262 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:39.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.418+0000 7f444f7fe640 10 monclient(hunting): get_auth_request con 0x7f4450142bf0 auth_method 0 2023-11-25T05:04:39.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.418+0000 7f444f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.418+0000 7f444f7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.418+0000 7f444f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.418+0000 7f632e7fc640 1 -- 172.21.15.7:0/3236682717 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f63340010c0 con 0x7f63380a8790 2023-11-25T05:04:39.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.418+0000 7f632e7fc640 10 client.4307.objecter ms_dispatch 0x7f6338000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:39.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.418+0000 7f632e7fc640 3 client.4307.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:39.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.418+0000 7f632e7fc640 3 client.4307.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:39.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.418+0000 7f632e7fc640 20 client.4307.objecter dump_active .. 0 homeless 2023-11-25T05:04:39.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.419+0000 7f632e7fc640 1 -- 172.21.15.7:0/3236682717 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6334057b80 con 0x7f63380a8790 2023-11-25T05:04:39.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.419+0000 7f632e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:39.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.419+0000 7f632e7fc640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:39.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.420+0000 7f444f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.420+0000 7f444f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.420+0000 7f444f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.423+0000 7f444f7fe640 10 monclient(hunting): handle_auth_done global_id 4350 payload 274 2023-11-25T05:04:39.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.423+0000 7f444f7fe640 1 -- 172.21.15.7:0/3301154262 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44500a8e10 msgr2=0x7f445013f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:39.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.423+0000 7f444f7fe640 1 --2- 172.21.15.7:0/3301154262 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44500a8e10 0x7f445013f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.423+0000 7f444f7fe640 1 -- 172.21.15.7:0/3301154262 >> v1:172.21.15.7:6789/0 conn(0x7f44500a8a30 legacy=0x7f44501424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.423+0000 7f444f7fe640 10 monclient: _finish_hunting 0 2023-11-25T05:04:39.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.423+0000 7f444f7fe640 1 monclient: found mon.noname-a 2023-11-25T05:04:39.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.423+0000 7f444f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:39.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.423+0000 7f444f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:39.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.423+0000 7f444f7fe640 1 -- 172.21.15.7:0/3301154262 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44500a97f0 con 0x7f4450142bf0 2023-11-25T05:04:39.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.429+0000 7f444f7fe640 1 --2- 172.21.15.7:0/3301154262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4450142bf0 0x7f445013ecf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4444009870 tx=0x7f444402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21362f88543a04 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f444effd640 1 -- 172.21.15.7:0/3301154262 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4444030840 con 0x7f4450142bf0 2023-11-25T05:04:39.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f444effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f444effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:39.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f444effd640 10 monclient: dump: 2023-11-25T05:04:39.511 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.511 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.511 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.511 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.511 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.512 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f444effd640 10 monclient: _finish_auth 0 2023-11-25T05:04:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f444effd640 10 monclient: _check_auth_tickets 2023-11-25T05:04:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f444effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f444effd640 1 -- 172.21.15.7:0/3301154262 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44440309a0 con 0x7f4450142bf0 2023-11-25T05:04:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f4456a0e640 5 monclient: authenticate success, global_id 4350 2023-11-25T05:04:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f4456a0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f444effd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:39.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f444effd640 1 -- 172.21.15.7:0/3301154262 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4444030d30 con 0x7f4450142bf0 2023-11-25T05:04:39.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f4456a0e640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:39.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f444effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f4456a0e640 4 set_mon_vals no callback set 2023-11-25T05:04:39.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f444effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:39.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.508+0000 7f444effd640 10 monclient: dump: 2023-11-25T05:04:39.513 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.514 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.514 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.514 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.514 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.514 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.514 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.515 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.515 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.515 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.509+0000 7f4456a0e640 10 monclient: shutdown 2023-11-25T05:04:39.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.509+0000 7f4456a0e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:39.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.509+0000 7f4456a0e640 1 -- 172.21.15.7:0/3301154262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4450142bf0 msgr2=0x7f445013ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.509+0000 7f4456a0e640 1 --2- 172.21.15.7:0/3301154262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4450142bf0 0x7f445013ecf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4444009870 tx=0x7f444402b980 comp rx=0 tx=0).stop 2023-11-25T05:04:39.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.509+0000 7f4456a0e640 1 -- 172.21.15.7:0/3301154262 shutdown_connections 2023-11-25T05:04:39.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.509+0000 7f4456a0e640 1 --2- 172.21.15.7:0/3301154262 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44500a8e10 0x7f445013f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.509+0000 7f4456a0e640 1 --2- 172.21.15.7:0/3301154262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4450142bf0 0x7f445013ecf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.509+0000 7f4456a0e640 1 -- 172.21.15.7:0/3301154262 >> 172.21.15.7:0/3301154262 conn(0x7f4450061a30 msgr2=0x7f44500b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:39.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.509+0000 7f4456a0e640 1 -- 172.21.15.7:0/3301154262 shutdown_connections 2023-11-25T05:04:39.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.509+0000 7f4456a0e640 1 -- 172.21.15.7:0/3301154262 wait complete. 2023-11-25T05:04:39.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.510+0000 7f8e17541640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-11-25T05:04:39.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.510+0000 7f8e17541640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-11-25T05:04:39.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.510+0000 7f8e17541640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:39.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.510+0000 7f4456a0e640 10 monclient: build_initial_monmap 2023-11-25T05:04:39.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.510+0000 7f8e17541640 1 -- 172.21.15.7:0/3918997230 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f8e10067ac0 con 0x7f8e10142bf0 2023-11-25T05:04:39.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.510+0000 7f4456a0e640 10 monclient: monmap: 2023-11-25T05:04:39.519 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:39.519 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:39.520 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:39.520 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:39.520 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:39.520 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.520 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:39.520 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:39.520 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:39.520 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.510+0000 7f4456a0e640 1 librados: starting msgr at 2023-11-25T05:04:39.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.510+0000 7f4456a0e640 1 librados: starting objecter 2023-11-25T05:04:39.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.510+0000 7f4456a0e640 1 Processor -- start 2023-11-25T05:04:39.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.510+0000 7f4456a0e640 1 -- start start 2023-11-25T05:04:39.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.510+0000 7f4456a0e640 1 librados: setting wanted keys 2023-11-25T05:04:39.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.510+0000 7f4456a0e640 1 librados: calling monclient init 2023-11-25T05:04:39.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.510+0000 7f4456a0e640 10 monclient: init 2023-11-25T05:04:39.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:39.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:04:39.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44500a8a30 0x7f4450138a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f44500a8a30 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:39.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44500a8e10 0x7f4450138f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f44500a8e10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:39.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4450142bf0 0x7f44500ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f4450142bf0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:39.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44500ab4f0 con 0x7f44500a8a30 2023-11-25T05:04:39.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44501394a0 con 0x7f4450142bf0 2023-11-25T05:04:39.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4450143eb0 con 0x7f44500a8e10 2023-11-25T05:04:39.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:39.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f4456a0e640 10 monclient(hunting): authenticate will time out at 1405.754209s 2023-11-25T05:04:39.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.511+0000 7f444ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44500a8a30 0x7f4450138a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.512+0000 7f444ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44500a8a30 0x7f4450138a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58486/0 (socket says 172.21.15.7:58486) 2023-11-25T05:04:39.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.512+0000 7f444ffff640 1 -- 172.21.15.7:0/473273627 learned_addr learned my addr 172.21.15.7:0/473273627 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:39.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.512+0000 7f444ffff640 10 monclient(hunting): get_auth_request con 0x7f44500a8a30 auth_method 0 2023-11-25T05:04:39.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.512+0000 7f444ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.512+0000 7f444ffff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.512+0000 7f444ffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.512+0000 7f444ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.512+0000 7f444ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.512+0000 7f444ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.515+0000 7f444ffff640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2023-11-25T05:04:39.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.515+0000 7f444ffff640 1 -- 172.21.15.7:0/473273627 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44500a8e10 msgr2=0x7f4450138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.515+0000 7f444ffff640 1 --2- 172.21.15.7:0/473273627 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44500a8e10 0x7f4450138f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.515+0000 7f444ffff640 1 -- 172.21.15.7:0/473273627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4450142bf0 msgr2=0x7f44500ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.515+0000 7f444ffff640 1 --2- 172.21.15.7:0/473273627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4450142bf0 0x7f44500ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.515+0000 7f444ffff640 10 monclient: _finish_hunting 0 2023-11-25T05:04:39.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.515+0000 7f444ffff640 1 monclient: found mon.noname-a 2023-11-25T05:04:39.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.515+0000 7f444ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:39.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.515+0000 7f444ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:39.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.515+0000 7f444ffff640 1 -- 172.21.15.7:0/473273627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44440094d0 con 0x7f44500a8a30 2023-11-25T05:04:39.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.517+0000 7f444ffff640 1 --2- 172.21.15.7:0/473273627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44500a8a30 0x7f4450138a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f444000c400 tx=0x7f444000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.517+0000 7f444cff9640 10 client.?.objecter ms_handle_connect 0x7f44500a8a30 2023-11-25T05:04:39.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.518+0000 7f444cff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:39.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f444cff9640 1 -- 172.21.15.7:0/473273627 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44400052d0 con 0x7f44500a8a30 2023-11-25T05:04:39.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f444cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f444cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:39.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f444cff9640 10 monclient: dump: 2023-11-25T05:04:39.528 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.528 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.528 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.528 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.528 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.528 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.529 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f444cff9640 10 monclient: _finish_auth 0 2023-11-25T05:04:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f444cff9640 10 monclient: _check_auth_tickets 2023-11-25T05:04:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f444cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f444cff9640 1 -- 172.21.15.7:0/473273627 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4440005450 con 0x7f44500a8a30 2023-11-25T05:04:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f4456a0e640 5 monclient: authenticate success, global_id 4353 2023-11-25T05:04:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f4456a0e640 10 monclient: _renew_subs 2023-11-25T05:04:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f4456a0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f4456a0e640 1 -- 172.21.15.7:0/473273627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4450145c40 con 0x7f44500a8a30 2023-11-25T05:04:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f444cff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f444cff9640 1 -- 172.21.15.7:0/473273627 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4440007260 con 0x7f44500a8a30 2023-11-25T05:04:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f444cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f444cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f444cff9640 10 monclient: dump: 2023-11-25T05:04:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.531 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.521+0000 7f444e7fc640 4 set_mon_vals no callback set 2023-11-25T05:04:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.522+0000 7f4456a0e640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:39.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.522+0000 7f4456a0e640 10 monclient: _renew_subs 2023-11-25T05:04:39.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.522+0000 7f4456a0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:39.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.522+0000 7f4456a0e640 1 -- 172.21.15.7:0/473273627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44500ac880 con 0x7f44500a8a30 2023-11-25T05:04:39.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.522+0000 7f4456a0e640 1 librados: init done 2023-11-25T05:04:39.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.522+0000 7f4456a0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:39.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.522+0000 7f4456a0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:39.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.522+0000 7f4456a0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:39.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.522+0000 7f4456a0e640 1 -- 172.21.15.7:0/473273627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4450067ac0 con 0x7f44500a8a30 2023-11-25T05:04:39.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.619+0000 7f633fe83640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-11-25T05:04:39.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.619+0000 7f633fe83640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-11-25T05:04:39.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.619+0000 7f633fe83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:39.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.619+0000 7f633fe83640 1 -- 172.21.15.7:0/3236682717 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f633813b4d0 con 0x7f63380a8790 2023-11-25T05:04:39.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.624+0000 7f632e7fc640 1 -- 172.21.15.7:0/3236682717 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6334057710 con 0x7f63380a8790 2023-11-25T05:04:39.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.624+0000 7f632e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-11-25T05:04:39.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.624+0000 7f632e7fc640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:39.627 INFO:teuthology.orchestra.run.smithi007.stdout:64424509445 2023-11-25T05:04:39.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 10 librados: watch_flush enter 2023-11-25T05:04:39.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 10 librados: watch_flush exit 2023-11-25T05:04:39.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 20 client.4307.objecter shutdown clearing up homeless session... 2023-11-25T05:04:39.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 10 client.4307.objecter successfully canceled tick 2023-11-25T05:04:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 1 -- 172.21.15.7:0/3236682717 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f631c06c820 msgr2=0x7f631c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 1 --2- 172.21.15.7:0/3236682717 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f631c06c820 0x7f631c06ece0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6328036000 tx=0x7f632803c040 comp rx=0 tx=0).stop 2023-11-25T05:04:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 10 monclient: shutdown 2023-11-25T05:04:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 1 -- 172.21.15.7:0/3236682717 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63380a8790 msgr2=0x7f633813bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 1 --2- 172.21.15.7:0/3236682717 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63380a8790 0x7f633813bf90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f633400c470 tx=0x7f633400c930 comp rx=0 tx=0).stop 2023-11-25T05:04:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 1 -- 172.21.15.7:0/3236682717 shutdown_connections 2023-11-25T05:04:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 1 --2- 172.21.15.7:0/3236682717 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f631c06c820 0x7f631c06ece0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 1 --2- 172.21.15.7:0/3236682717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63380a9bf0 0x7f63381316e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 1 --2- 172.21.15.7:0/3236682717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63380a8b70 0x7f6338139a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 1 --2- 172.21.15.7:0/3236682717 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63380a8790 0x7f633813bf90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 1 -- 172.21.15.7:0/3236682717 >> 172.21.15.7:0/3236682717 conn(0x7f63380b41f0 msgr2=0x7f63381405e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 1 -- 172.21.15.7:0/3236682717 shutdown_connections 2023-11-25T05:04:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 1 -- 172.21.15.7:0/3236682717 wait complete. 2023-11-25T05:04:39.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.626+0000 7f633fe83640 1 librados: shutdown 2023-11-25T05:04:39.639 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.6 2023-11-25T05:04:39.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.675+0000 7f1196ffd640 1 -- 172.21.15.7:0/3568551283 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f119c054c50 con 0x7f11a8141f90 2023-11-25T05:04:39.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.675+0000 7f1196ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:39.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.675+0000 7f1196ffd640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:39.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.729+0000 7f8e05ffb640 1 -- 172.21.15.7:0/3918997230 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8e08043090 con 0x7f8e10142bf0 2023-11-25T05:04:39.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.729+0000 7f8e05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-11-25T05:04:39.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.729+0000 7f8e05ffb640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:39.731 INFO:teuthology.orchestra.run.smithi007.stdout:68719476740 2023-11-25T05:04:39.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.730+0000 7f8e17541640 10 librados: watch_flush enter 2023-11-25T05:04:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.730+0000 7f8e17541640 10 librados: watch_flush exit 2023-11-25T05:04:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.730+0000 7f8e17541640 20 client.4366.objecter shutdown clearing up homeless session... 2023-11-25T05:04:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.730+0000 7f8e17541640 10 client.4366.objecter successfully canceled tick 2023-11-25T05:04:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.731+0000 7f8e17541640 1 -- 172.21.15.7:0/3918997230 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f8df406c7d0 msgr2=0x7f8df406ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.731+0000 7f8e17541640 1 --2- 172.21.15.7:0/3918997230 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f8df406c7d0 0x7f8df406ec90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c004f60 tx=0x7f8e0c007e30 comp rx=0 tx=0).stop 2023-11-25T05:04:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.731+0000 7f8e17541640 10 monclient: shutdown 2023-11-25T05:04:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.731+0000 7f8e17541640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.731+0000 7f8e17541640 1 -- 172.21.15.7:0/3918997230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e10142bf0 msgr2=0x7f8e100b0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.731+0000 7f8e17541640 1 --2- 172.21.15.7:0/3918997230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e10142bf0 0x7f8e100b0a90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8e08009f40 tx=0x7f8e08004a40 comp rx=0 tx=0).stop 2023-11-25T05:04:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.731+0000 7f8e17541640 1 -- 172.21.15.7:0/3918997230 shutdown_connections 2023-11-25T05:04:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.731+0000 7f8e17541640 1 --2- 172.21.15.7:0/3918997230 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f8df406c7d0 0x7f8df406ec90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.731+0000 7f8e17541640 1 --2- 172.21.15.7:0/3918997230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e10142bf0 0x7f8e100b0a90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.731+0000 7f8e17541640 1 --2- 172.21.15.7:0/3918997230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e100a8e10 0x7f8e100b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.731+0000 7f8e17541640 1 --2- 172.21.15.7:0/3918997230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8e100a8a30 0x7f8e100ac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.731+0000 7f8e17541640 1 -- 172.21.15.7:0/3918997230 >> 172.21.15.7:0/3918997230 conn(0x7f8e10061a30 msgr2=0x7f8e1013cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:39.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.731+0000 7f8e17541640 1 -- 172.21.15.7:0/3918997230 shutdown_connections 2023-11-25T05:04:39.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.731+0000 7f8e17541640 1 -- 172.21.15.7:0/3918997230 wait complete. 2023-11-25T05:04:39.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.731+0000 7f8e17541640 1 librados: shutdown 2023-11-25T05:04:39.744 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.0 2023-11-25T05:04:39.777 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-25T05:04:39.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.792+0000 7f444cff9640 1 -- 172.21.15.7:0/473273627 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f44400073c0 con 0x7f44500a8a30 2023-11-25T05:04:39.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.792+0000 7f444cff9640 1 --2- 172.21.15.7:0/473273627 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f443c06c820 0x7f443c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.793+0000 7f444f7fe640 1 --2- 172.21.15.7:0/473273627 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f443c06c820 0x7f443c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.793+0000 7f444f7fe640 10 monclient: get_auth_request con 0x7f443c06c820 auth_method 0 2023-11-25T05:04:39.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.793+0000 7f444cff9640 10 client.4353.objecter ms_handle_connect 0x7f443c06c820 2023-11-25T05:04:39.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.793+0000 7f444f7fe640 1 --2- 172.21.15.7:0/473273627 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f443c06c820 0x7f443c06ece0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4444036000 tx=0x7f4444030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.805+0000 7f444cff9640 1 -- 172.21.15.7:0/473273627 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4440007670 con 0x7f44500a8a30 2023-11-25T05:04:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.805+0000 7f444cff9640 10 client.4353.objecter ms_dispatch 0x7f4450000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.805+0000 7f444cff9640 3 client.4353.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.805+0000 7f444cff9640 3 client.4353.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.806+0000 7f444cff9640 20 client.4353.objecter dump_active .. 0 homeless 2023-11-25T05:04:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.806+0000 7f444cff9640 1 -- 172.21.15.7:0/473273627 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4440053fd0 con 0x7f44500a8a30 2023-11-25T05:04:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.806+0000 7f444cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.806+0000 7f444cff9640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:39.817 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-25T05:04:39.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.871+0000 7f11acc21640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-11-25T05:04:39.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.871+0000 7f11acc21640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-11-25T05:04:39.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.871+0000 7f11acc21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:39.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.871+0000 7f11acc21640 1 -- 172.21.15.7:0/3568551283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f11a81c80b0 con 0x7f11a8141f90 2023-11-25T05:04:39.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.875+0000 7f1196ffd640 1 -- 172.21.15.7:0/3568551283 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f119c021090 con 0x7f11a8141f90 2023-11-25T05:04:39.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.875+0000 7f1196ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-11-25T05:04:39.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.875+0000 7f1196ffd640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:39.877 INFO:teuthology.orchestra.run.smithi007.stdout:68719476740 2023-11-25T05:04:39.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 10 librados: watch_flush enter 2023-11-25T05:04:39.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 10 librados: watch_flush exit 2023-11-25T05:04:39.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 20 client.4372.objecter shutdown clearing up homeless session... 2023-11-25T05:04:39.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 10 client.4372.objecter successfully canceled tick 2023-11-25T05:04:39.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 1 -- 172.21.15.7:0/3568551283 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f118806c7d0 msgr2=0x7f118806ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 1 --2- 172.21.15.7:0/3568551283 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f118806c7d0 0x7f118806ec90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1198009990 tx=0x7f1198009110 comp rx=0 tx=0).stop 2023-11-25T05:04:39.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 10 monclient: shutdown 2023-11-25T05:04:39.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:39.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 1 -- 172.21.15.7:0/3568551283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a8141f90 msgr2=0x7f11a80ae470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 1 --2- 172.21.15.7:0/3568551283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a8141f90 0x7f11a80ae470 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f119c00e9e0 tx=0x7f119c00ede0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 1 -- 172.21.15.7:0/3568551283 shutdown_connections 2023-11-25T05:04:39.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 1 --2- 172.21.15.7:0/3568551283 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f118806c7d0 0x7f118806ec90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 1 --2- 172.21.15.7:0/3568551283 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11a813a6a0 0x7f11a80b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 1 --2- 172.21.15.7:0/3568551283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a8141f90 0x7f11a80ae470 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 1 --2- 172.21.15.7:0/3568551283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11a80a8910 0x7f11a81c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 1 -- 172.21.15.7:0/3568551283 >> 172.21.15.7:0/3568551283 conn(0x7f11a80b7780 msgr2=0x7f11a813c860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:39.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 1 -- 172.21.15.7:0/3568551283 shutdown_connections 2023-11-25T05:04:39.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 1 -- 172.21.15.7:0/3568551283 wait complete. 2023-11-25T05:04:39.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.877+0000 7f11acc21640 1 librados: shutdown 2023-11-25T05:04:39.892 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.2 2023-11-25T05:04:39.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.919+0000 7f6a6fe51640 10 monclient: get_monmap_and_config 2023-11-25T05:04:39.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.919+0000 7f6a6fe51640 10 monclient: build_initial_monmap 2023-11-25T05:04:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.919+0000 7f6a6fe51640 10 monclient: monmap: 2023-11-25T05:04:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:39.920069+0000 2023-11-25T05:04:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:39.920069+0000 2023-11-25T05:04:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:39.922 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.919+0000 7f6a6fe51640 1 Processor -- start 2023-11-25T05:04:39.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.920+0000 7f6a6fe51640 1 -- start start 2023-11-25T05:04:39.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.920+0000 7f6a6fe51640 10 monclient: init 2023-11-25T05:04:39.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.921+0000 7f6a6fe51640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:39.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.921+0000 7f6a6fe51640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-11-25T05:04:39.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.921+0000 7f6a6fe51640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a680a8910 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:39.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.921+0000 7f6a6fe51640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6a680a9b70 0x7f6a6813f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.921+0000 7f6a6fe51640 10 monclient(hunting): picked mon.noname-a con 0x7f6a680a9b70 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:39.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.921+0000 7f6a6fe51640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a68141f90 0x7f6a681423f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.921+0000 7f6a6fe51640 10 monclient(hunting): picked mon.noname-b con 0x7f6a68141f90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:39.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.921+0000 7f6a6fe51640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a680a8cf0 con 0x7f6a680a8910 2023-11-25T05:04:39.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.921+0000 7f6a6fe51640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.921+0000 7f6a6fe51640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a680aabd0 con 0x7f6a680a9b70 2023-11-25T05:04:39.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.921+0000 7f6a6fe51640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.921+0000 7f6a6fe51640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680aaeb0 con 0x7f6a68141f90 2023-11-25T05:04:39.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.921+0000 7f6a6fe51640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:39.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.921+0000 7f6a6fe51640 10 monclient(hunting): authenticate will time out at 1406.164148s 2023-11-25T05:04:39.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.922+0000 7f6a6e3ee640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a68141f90 0x7f6a681423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.922+0000 7f6a6e3ee640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a68141f90 0x7f6a681423f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41922/0 (socket says 172.21.15.7:41922) 2023-11-25T05:04:39.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.922+0000 7f6a6e3ee640 1 -- 172.21.15.7:0/3957474331 learned_addr learned my addr 172.21.15.7:0/3957474331 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:39.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.922+0000 7f6a6e3ee640 10 monclient(hunting): get_auth_request con 0x7f6a68141f90 auth_method 0 2023-11-25T05:04:39.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.922+0000 7f6a6e3ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.922+0000 7f6a6e3ee640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.922+0000 7f6a6e3ee640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.923+0000 7f6a6e3ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.923+0000 7f6a6e3ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.923+0000 7f6a6e3ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.925+0000 7f6a6e3ee640 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2023-11-25T05:04:39.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.925+0000 7f6a6e3ee640 1 -- 172.21.15.7:0/3957474331 >> v2:172.21.15.7:3300/0 conn(0x7f6a680a9b70 msgr2=0x7f6a6813f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:39.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.925+0000 7f6a6e3ee640 1 --2- 172.21.15.7:0/3957474331 >> v2:172.21.15.7:3300/0 conn(0x7f6a680a9b70 0x7f6a6813f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.925+0000 7f6a6e3ee640 1 -- 172.21.15.7:0/3957474331 >> v1:172.21.15.7:6789/0 conn(0x7f6a680a8910 legacy=0x7f6a6813ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:39.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.925+0000 7f6a6e3ee640 10 monclient: _finish_hunting 0 2023-11-25T05:04:39.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.925+0000 7f6a6e3ee640 1 monclient: found mon.noname-b 2023-11-25T05:04:39.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.925+0000 7f6a6e3ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:39.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.925+0000 7f6a6e3ee640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:04:39.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.925+0000 7f6a6e3ee640 1 -- 172.21.15.7:0/3957474331 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a680a9720 con 0x7f6a68141f90 2023-11-25T05:04:39.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.928+0000 7f6a6e3ee640 1 --2- 172.21.15.7:0/3957474331 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a68141f90 0x7f6a681423f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a68074590 tx=0x7f6a6002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99530d79d1eb046 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.931+0000 7fdf212af640 10 monclient: get_monmap_and_config 2023-11-25T05:04:39.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.931+0000 7fdf212af640 10 monclient: build_initial_monmap 2023-11-25T05:04:39.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.931+0000 7fdf212af640 10 monclient: monmap: 2023-11-25T05:04:39.933 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:39.933 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.933 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:39.932533+0000 2023-11-25T05:04:39.933 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:39.932533+0000 2023-11-25T05:04:39.933 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:39.933 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.933 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:39.933 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:39.934 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:39.934 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:39.934 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:39.934 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.932+0000 7fdf212af640 1 Processor -- start 2023-11-25T05:04:39.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.932+0000 7fdf212af640 1 -- start start 2023-11-25T05:04:39.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.932+0000 7fdf212af640 10 monclient: init 2023-11-25T05:04:39.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.932+0000 7fdf212af640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:39.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.932+0000 7fdf212af640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-11-25T05:04:39.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.932+0000 7fdf212af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf1c0a8a30 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:39.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.932+0000 7fdf212af640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdf1c142bf0 0x7fdf1c13ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.932+0000 7fdf212af640 10 monclient(hunting): picked mon.noname-b con 0x7fdf1c142bf0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.932+0000 7fdf212af640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdf1c0a8e10 0x7fdf1c13f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.933+0000 7fdf212af640 10 monclient(hunting): picked mon.noname-c con 0x7fdf1c0a8e10 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.933+0000 7fdf212af640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf1c0aac10 con 0x7fdf1c0a8a30 2023-11-25T05:04:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.933+0000 7fdf212af640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.933+0000 7fdf212af640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf1c0aaa10 con 0x7fdf1c0a8e10 2023-11-25T05:04:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.933+0000 7fdf212af640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.933+0000 7fdf212af640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c0aaef0 con 0x7fdf1c142bf0 2023-11-25T05:04:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.933+0000 7fdf212af640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.933+0000 7fdf212af640 10 monclient(hunting): authenticate will time out at 1406.175989s 2023-11-25T05:04:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.933+0000 7fdf1b7fe640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdf1c0a8e10 0x7fdf1c13f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7fdf1b7fe640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdf1c0a8e10 0x7fdf1c13f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:35278/0 (socket says 172.21.15.7:35278) 2023-11-25T05:04:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7fdf1b7fe640 1 -- 172.21.15.7:0/2360216557 learned_addr learned my addr 172.21.15.7:0/2360216557 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7fdf1b7fe640 10 monclient(hunting): get_auth_request con 0x7fdf1c0a8e10 auth_method 0 2023-11-25T05:04:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7fdf1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7fdf1b7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7fdf1b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6cbeb640 1 -- 172.21.15.7:0/3957474331 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6002ca50 con 0x7f6a68141f90 2023-11-25T05:04:39.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6cbeb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6cbeb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:39.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6cbeb640 10 monclient: dump: 2023-11-25T05:04:39.940 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.940 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.940 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.940 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.940 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.940 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.940 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.941 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.941 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.941 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6cbeb640 10 monclient: _finish_auth 0 2023-11-25T05:04:39.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6cbeb640 10 monclient: _check_auth_tickets 2023-11-25T05:04:39.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6cbeb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:39.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6cbeb640 1 -- 172.21.15.7:0/3957474331 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6002cbb0 con 0x7f6a68141f90 2023-11-25T05:04:39.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6fe51640 5 monclient: authenticate success, global_id 4310 2023-11-25T05:04:39.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6fe51640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:39.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6cbeb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:39.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6fe51640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:39.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6cbeb640 1 -- 172.21.15.7:0/3957474331 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a600343f0 con 0x7f6a68141f90 2023-11-25T05:04:39.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6cbeb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6fe51640 4 set_mon_vals no callback set 2023-11-25T05:04:39.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6cbeb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:39.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.934+0000 7f6a6cbeb640 10 monclient: dump: 2023-11-25T05:04:39.942 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.942 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.942 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.942 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.943 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.943 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.943 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.943 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.943 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.943 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7fdf1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7fdf1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7fdf1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7fdf1a7fc640 1 --2- 172.21.15.7:0/2360216557 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdf1c142bf0 0x7fdf1c13ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7f6a6fe51640 10 monclient: shutdown 2023-11-25T05:04:39.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7f6a6fe51640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:39.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7f6a6fe51640 1 -- 172.21.15.7:0/3957474331 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a68141f90 msgr2=0x7f6a681423f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7f6a6fe51640 1 --2- 172.21.15.7:0/3957474331 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a68141f90 0x7f6a681423f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a68074590 tx=0x7f6a6002beb0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7f6a6fe51640 1 -- 172.21.15.7:0/3957474331 shutdown_connections 2023-11-25T05:04:39.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7f6a6fe51640 1 --2- 172.21.15.7:0/3957474331 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a68141f90 0x7f6a681423f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7f6a6fe51640 1 --2- 172.21.15.7:0/3957474331 >> v2:172.21.15.7:3300/0 conn(0x7f6a680a9b70 0x7f6a6813f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7f6a6fe51640 1 -- 172.21.15.7:0/3957474331 >> 172.21.15.7:0/3957474331 conn(0x7f6a680b7780 msgr2=0x7f6a680b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:39.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7f6a6fe51640 1 -- 172.21.15.7:0/3957474331 shutdown_connections 2023-11-25T05:04:39.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7f6a6fe51640 1 -- 172.21.15.7:0/3957474331 wait complete. 2023-11-25T05:04:39.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7fdf1a7fc640 10 monclient(hunting): get_auth_request con 0x7fdf1c142bf0 auth_method 0 2023-11-25T05:04:39.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7fdf1a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7fdf1a7fc640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.935+0000 7fdf1a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.936+0000 7f6a6fe51640 10 monclient: build_initial_monmap 2023-11-25T05:04:39.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.936+0000 7f6a6fe51640 10 monclient: monmap: 2023-11-25T05:04:39.948 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:39.949 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.936+0000 7fdf1a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.936+0000 7fdf1a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.936+0000 7fdf1a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.936+0000 7f6a6fe51640 1 librados: starting msgr at 2023-11-25T05:04:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.936+0000 7f6a6fe51640 1 librados: starting objecter 2023-11-25T05:04:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 1 Processor -- start 2023-11-25T05:04:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 1 -- start start 2023-11-25T05:04:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 1 librados: setting wanted keys 2023-11-25T05:04:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 1 librados: calling monclient init 2023-11-25T05:04:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 10 monclient: init 2023-11-25T05:04:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:04:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a680a8910 0x7f6a680ab540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 10 monclient(hunting): picked mon.noname-a con 0x7f6a680a8910 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a68141f90 0x7f6a68138f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 10 monclient(hunting): picked mon.noname-c con 0x7f6a68141f90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a680ac070 0x7f6a68139480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 10 monclient(hunting): picked mon.noname-b con 0x7f6a680ac070 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680abf10 con 0x7f6a680a8910 2023-11-25T05:04:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680ad470 con 0x7f6a680ac070 2023-11-25T05:04:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681399c0 con 0x7f6a68141f90 2023-11-25T05:04:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.937+0000 7f6a6fe51640 10 monclient(hunting): authenticate will time out at 1406.180578s 2023-11-25T05:04:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.938+0000 7fdf1b7fe640 10 monclient(hunting): handle_auth_done global_id 4375 payload 274 2023-11-25T05:04:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.938+0000 7fdf1b7fe640 1 -- 172.21.15.7:0/2360216557 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdf1c142bf0 msgr2=0x7fdf1c13ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.938+0000 7fdf1b7fe640 1 --2- 172.21.15.7:0/2360216557 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdf1c142bf0 0x7fdf1c13ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.938+0000 7fdf1b7fe640 1 -- 172.21.15.7:0/2360216557 >> v1:172.21.15.90:6789/0 conn(0x7fdf1c0a8a30 legacy=0x7fdf1c1424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.938+0000 7fdf1b7fe640 10 monclient: _finish_hunting 0 2023-11-25T05:04:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.938+0000 7fdf1b7fe640 1 monclient: found mon.noname-c 2023-11-25T05:04:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.938+0000 7fdf1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.938+0000 7fdf1b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.938+0000 7fdf1b7fe640 1 -- 172.21.15.7:0/2360216557 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c0a97f0 con 0x7fdf1c0a8e10 2023-11-25T05:04:39.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-25T05:04:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.940+0000 7f6a6e3ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a680ac070 0x7f6a68139480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.940+0000 7fdf1b7fe640 1 --2- 172.21.15.7:0/2360216557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf1c0a8e10 0x7fdf1c13f230 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf14009e50 tx=0x7fdf1402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d6a5931ec64cb77 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.943+0000 7f6a6d3ec640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a68141f90 0x7f6a68138f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.944+0000 7f6a6d3ec640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a68141f90 0x7f6a68138f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41936/0 (socket says 172.21.15.7:41936) 2023-11-25T05:04:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.944+0000 7f6a6e3ee640 10 monclient(hunting): get_auth_request con 0x7f6a680ac070 auth_method 0 2023-11-25T05:04:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.944+0000 7f6a6d3ec640 1 -- 172.21.15.7:0/3478383938 learned_addr learned my addr 172.21.15.7:0/3478383938 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.944+0000 7f6a6e3ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.944+0000 7f6a6e3ee640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.944+0000 7f6a6e3ee640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.944+0000 7f6a6d3ec640 10 monclient(hunting): get_auth_request con 0x7f6a68141f90 auth_method 0 2023-11-25T05:04:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.944+0000 7f6a6d3ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.944+0000 7f6a6d3ec640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.944+0000 7f6a6d3ec640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7f6a6d3ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7f6a6d3ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7f6a6d3ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf19ffb640 1 -- 172.21.15.7:0/2360216557 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1402cb00 con 0x7fdf1c0a8e10 2023-11-25T05:04:39.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf19ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:39.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf19ffb640 10 monclient: dump: 2023-11-25T05:04:39.958 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.958 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.958 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.959 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.959 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.959 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.959 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.959 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.959 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.959 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf19ffb640 10 monclient: _finish_auth 0 2023-11-25T05:04:39.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf19ffb640 10 monclient: _check_auth_tickets 2023-11-25T05:04:39.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:39.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf19ffb640 1 -- 172.21.15.7:0/2360216557 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1402cc60 con 0x7fdf1c0a8e10 2023-11-25T05:04:39.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf212af640 5 monclient: authenticate success, global_id 4375 2023-11-25T05:04:39.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf19ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:39.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf212af640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:39.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf19ffb640 1 -- 172.21.15.7:0/2360216557 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf140343f0 con 0x7fdf1c0a8e10 2023-11-25T05:04:39.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf212af640 4 set_mon_vals no callback set 2023-11-25T05:04:39.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:39.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7fdf19ffb640 10 monclient: dump: 2023-11-25T05:04:39.961 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.961 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.962 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.962 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.962 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.962 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.962 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.962 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.962 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.962 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7f6a6e3ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7f6a6e3ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.945+0000 7f6a6e3ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.946+0000 7fdf212af640 10 monclient: shutdown 2023-11-25T05:04:39.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.946+0000 7fdf212af640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:39.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.946+0000 7fdf212af640 1 -- 172.21.15.7:0/2360216557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf1c0a8e10 msgr2=0x7fdf1c13f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.946+0000 7fdf212af640 1 --2- 172.21.15.7:0/2360216557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf1c0a8e10 0x7fdf1c13f230 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf14009e50 tx=0x7fdf1402bf60 comp rx=0 tx=0).stop 2023-11-25T05:04:39.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.946+0000 7fdf212af640 1 -- 172.21.15.7:0/2360216557 shutdown_connections 2023-11-25T05:04:39.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.946+0000 7fdf212af640 1 --2- 172.21.15.7:0/2360216557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf1c0a8e10 0x7fdf1c13f230 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.946+0000 7fdf212af640 1 --2- 172.21.15.7:0/2360216557 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdf1c142bf0 0x7fdf1c13ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.946+0000 7fdf212af640 1 -- 172.21.15.7:0/2360216557 >> 172.21.15.7:0/2360216557 conn(0x7fdf1c061a30 msgr2=0x7fdf1c0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:39.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.946+0000 7fdf212af640 1 -- 172.21.15.7:0/2360216557 shutdown_connections 2023-11-25T05:04:39.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.946+0000 7fdf212af640 1 -- 172.21.15.7:0/2360216557 wait complete. 2023-11-25T05:04:39.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.947+0000 7fdf212af640 10 monclient: build_initial_monmap 2023-11-25T05:04:39.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.947+0000 7fdf212af640 10 monclient: monmap: 2023-11-25T05:04:39.965 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:39.966 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:39.966 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:39.966 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:39.966 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:39.966 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.966 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:39.966 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:39.966 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:39.966 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.947+0000 7f6a6dbed640 1 --2- 172.21.15.7:0/3478383938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a680a8910 0x7f6a680ab540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.947+0000 7fdf212af640 1 librados: starting msgr at 2023-11-25T05:04:39.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.947+0000 7fdf212af640 1 librados: starting objecter 2023-11-25T05:04:39.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.948+0000 7fdf212af640 1 Processor -- start 2023-11-25T05:04:39.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.948+0000 7fdf212af640 1 -- start start 2023-11-25T05:04:39.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.948+0000 7fdf212af640 1 librados: setting wanted keys 2023-11-25T05:04:39.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.948+0000 7fdf212af640 1 librados: calling monclient init 2023-11-25T05:04:39.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.948+0000 7fdf212af640 10 monclient: init 2023-11-25T05:04:39.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.948+0000 7f6a6e3ee640 10 monclient(hunting): handle_auth_done global_id 4378 payload 824 2023-11-25T05:04:39.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.948+0000 7f6a6e3ee640 1 -- 172.21.15.7:0/3478383938 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a68141f90 msgr2=0x7f6a68138f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.948+0000 7f6a6e3ee640 1 --2- 172.21.15.7:0/3478383938 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a68141f90 0x7f6a68138f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.948+0000 7f6a6e3ee640 1 -- 172.21.15.7:0/3478383938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a680a8910 msgr2=0x7f6a680ab540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.948+0000 7f6a6e3ee640 1 --2- 172.21.15.7:0/3478383938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a680a8910 0x7f6a680ab540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.948+0000 7f6a6e3ee640 10 monclient: _finish_hunting 0 2023-11-25T05:04:39.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.948+0000 7f6a6e3ee640 1 monclient: found mon.noname-b 2023-11-25T05:04:39.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.948+0000 7f6a6e3ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:39.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.948+0000 7f6a6e3ee640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:39.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.948+0000 7f6a6e3ee640 1 -- 172.21.15.7:0/3478383938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a60009b30 con 0x7f6a680ac070 2023-11-25T05:04:39.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:39.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:04:39.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf1c0a8a30 0x7fdf1c0abf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 10 monclient(hunting): picked mon.noname-a con 0x7fdf1c0a8a30 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:39.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdf1c0a8e10 0x7fdf1c0ac450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 10 monclient(hunting): picked mon.noname-c con 0x7fdf1c0a8e10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:39.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf1c142bf0 0x7fdf1c144580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 10 monclient(hunting): picked mon.noname-b con 0x7fdf1c142bf0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:39.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c1440f0 con 0x7fdf1c0a8a30 2023-11-25T05:04:39.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c143cf0 con 0x7fdf1c142bf0 2023-11-25T05:04:39.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:39.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c0ac990 con 0x7fdf1c0a8e10 2023-11-25T05:04:39.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:39.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.949+0000 7fdf212af640 10 monclient(hunting): authenticate will time out at 1406.192108s 2023-11-25T05:04:39.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.952+0000 7f6a6e3ee640 1 --2- 172.21.15.7:0/3478383938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a680ac070 0x7f6a68139480 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6a60032000 tx=0x7f6a60004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.952+0000 7f6a5e7fc640 10 client.?.objecter ms_handle_connect 0x7f6a680ac070 2023-11-25T05:04:39.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.952+0000 7f6a5e7fc640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:39.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.954+0000 7fdf1affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf1c0a8a30 0x7fdf1c0abf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.955+0000 7fdf1affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf1c0a8a30 0x7fdf1c0abf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58500/0 (socket says 172.21.15.7:58500) 2023-11-25T05:04:39.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.955+0000 7fdf1affd640 1 -- 172.21.15.7:0/1947578733 learned_addr learned my addr 172.21.15.7:0/1947578733 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:39.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.955+0000 7fdf1affd640 10 monclient(hunting): get_auth_request con 0x7fdf1c0a8a30 auth_method 0 2023-11-25T05:04:39.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.955+0000 7fdf1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:39.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.955+0000 7fdf1affd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:39.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.955+0000 7fdf1affd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:39.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.955+0000 7fdf1affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:39.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.955+0000 7fdf1affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:39.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.955+0000 7fdf1affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:39.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.958+0000 7fdf1affd640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2023-11-25T05:04:39.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.958+0000 7fdf1affd640 1 -- 172.21.15.7:0/1947578733 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdf1c0a8e10 msgr2=0x7fdf1c0ac450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.958+0000 7fdf1affd640 1 --2- 172.21.15.7:0/1947578733 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdf1c0a8e10 0x7fdf1c0ac450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.958+0000 7fdf1affd640 1 -- 172.21.15.7:0/1947578733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf1c142bf0 msgr2=0x7fdf1c144580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:39.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.958+0000 7fdf1affd640 1 --2- 172.21.15.7:0/1947578733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf1c142bf0 0x7fdf1c144580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:39.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.958+0000 7fdf1affd640 10 monclient: _finish_hunting 0 2023-11-25T05:04:39.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.958+0000 7fdf1affd640 1 monclient: found mon.noname-a 2023-11-25T05:04:39.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.958+0000 7fdf1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:39.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.958+0000 7fdf1affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:39.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.958+0000 7fdf1affd640 1 -- 172.21.15.7:0/1947578733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf14009b30 con 0x7fdf1c0a8a30 2023-11-25T05:04:39.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.962+0000 7fdf1affd640 1 --2- 172.21.15.7:0/1947578733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf1c0a8a30 0x7fdf1c0abf10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c00c450 tx=0x7fdf0c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:39.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.962+0000 7fdf037fe640 10 client.?.objecter ms_handle_connect 0x7fdf1c0a8a30 2023-11-25T05:04:39.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.962+0000 7fdf037fe640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:39.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf037fe640 1 -- 172.21.15.7:0/1947578733 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c0052d0 con 0x7fdf1c0a8a30 2023-11-25T05:04:39.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:39.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf037fe640 10 monclient: dump: 2023-11-25T05:04:39.980 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.980 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.980 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.980 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.980 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.980 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.980 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.980 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.980 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.980 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf037fe640 10 monclient: _finish_auth 0 2023-11-25T05:04:39.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf037fe640 10 monclient: _check_auth_tickets 2023-11-25T05:04:39.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:39.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf037fe640 1 -- 172.21.15.7:0/1947578733 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0c005430 con 0x7fdf1c0a8a30 2023-11-25T05:04:39.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf037fe640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:39.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf037fe640 1 -- 172.21.15.7:0/1947578733 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c003830 con 0x7fdf1c0a8a30 2023-11-25T05:04:39.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:39.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:39.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf037fe640 10 monclient: dump: 2023-11-25T05:04:39.983 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:39.983 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:39.983 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.983 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:39.983 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:39.983 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:39.983 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:39.983 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:39.983 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:39.984 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:39.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf212af640 5 monclient: authenticate success, global_id 4356 2023-11-25T05:04:39.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf212af640 10 monclient: _renew_subs 2023-11-25T05:04:39.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf212af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:39.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf197fa640 4 set_mon_vals no callback set 2023-11-25T05:04:39.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf212af640 1 -- 172.21.15.7:0/1947578733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1c144ac0 con 0x7fdf1c0a8a30 2023-11-25T05:04:39.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf212af640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:39.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf212af640 10 monclient: _renew_subs 2023-11-25T05:04:39.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf212af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:39.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf212af640 1 -- 172.21.15.7:0/1947578733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1c144f80 con 0x7fdf1c0a8a30 2023-11-25T05:04:39.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.967+0000 7fdf212af640 1 librados: init done 2023-11-25T05:04:39.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.968+0000 7fdf212af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:39.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.968+0000 7fdf212af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:39.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.968+0000 7fdf212af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:39.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.968+0000 7fdf212af640 1 -- 172.21.15.7:0/1947578733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1c067ac0 con 0x7fdf1c0a8a30 2023-11-25T05:04:39.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.996+0000 7fdf037fe640 1 -- 172.21.15.7:0/1947578733 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdf0c003990 con 0x7fdf1c0a8a30 2023-11-25T05:04:39.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.997+0000 7fdf037fe640 1 --2- 172.21.15.7:0/1947578733 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdf0806c7d0 0x7fdf0806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:39.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.997+0000 7fdf1a7fc640 1 --2- 172.21.15.7:0/1947578733 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdf0806c7d0 0x7fdf0806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:39.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.997+0000 7fdf1a7fc640 10 monclient: get_auth_request con 0x7fdf0806c7d0 auth_method 0 2023-11-25T05:04:40.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.998+0000 7fdf037fe640 10 client.4356.objecter ms_handle_connect 0x7fdf0806c7d0 2023-11-25T05:04:40.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.998+0000 7fdf1a7fc640 1 --2- 172.21.15.7:0/1947578733 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdf0806c7d0 0x7fdf0806ec90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdf10006a90 tx=0x7fdf10006a20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:40.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.999+0000 7fdf037fe640 1 -- 172.21.15.7:0/1947578733 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdf0c0584c0 con 0x7fdf1c0a8a30 2023-11-25T05:04:40.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.999+0000 7fdf037fe640 10 client.4356.objecter ms_dispatch 0x7fdf1c000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:40.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.999+0000 7fdf037fe640 3 client.4356.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:40.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.999+0000 7fdf037fe640 3 client.4356.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:40.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:39.999+0000 7fdf037fe640 20 client.4356.objecter dump_active .. 0 homeless 2023-11-25T05:04:40.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a5e7fc640 1 -- 172.21.15.7:0/3478383938 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60040670 con 0x7f6a680ac070 2023-11-25T05:04:40.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a5e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:40.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a5e7fc640 10 monclient: dump: 2023-11-25T05:04:40.011 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.011 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.011 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.011 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.011 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.011 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.011 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.011 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.012 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.012 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a5e7fc640 10 monclient: _finish_auth 0 2023-11-25T05:04:40.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a5e7fc640 10 monclient: _check_auth_tickets 2023-11-25T05:04:40.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:40.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a5e7fc640 1 -- 172.21.15.7:0/3478383938 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a600407f0 con 0x7f6a680ac070 2023-11-25T05:04:40.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a6fe51640 5 monclient: authenticate success, global_id 4378 2023-11-25T05:04:40.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a6fe51640 10 monclient: _renew_subs 2023-11-25T05:04:40.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a6fe51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:40.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a6fe51640 1 -- 172.21.15.7:0/3478383938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6813ba90 con 0x7f6a680ac070 2023-11-25T05:04:40.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a5e7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:40.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a5e7fc640 1 -- 172.21.15.7:0/3478383938 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60040ba0 con 0x7f6a680ac070 2023-11-25T05:04:40.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:40.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a5e7fc640 10 monclient: dump: 2023-11-25T05:04:40.014 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.014 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.014 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.014 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.014 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.014 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.014 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.014 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.014 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.014 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a5ffff640 4 set_mon_vals no callback set 2023-11-25T05:04:40.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a6fe51640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:40.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a6fe51640 10 monclient: _renew_subs 2023-11-25T05:04:40.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a6fe51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:40.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.008+0000 7f6a6fe51640 1 -- 172.21.15.7:0/3478383938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a680aba80 con 0x7f6a680ac070 2023-11-25T05:04:40.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.009+0000 7f6a6fe51640 1 librados: init done 2023-11-25T05:04:40.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.009+0000 7f6a6fe51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:40.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.009+0000 7f6a6fe51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:40.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.009+0000 7f6a6fe51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:40.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.009+0000 7f6a6fe51640 1 -- 172.21.15.7:0/3478383938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6813b790 con 0x7f6a680ac070 2023-11-25T05:04:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.031+0000 7f4456a0e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-11-25T05:04:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.031+0000 7f4456a0e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-11-25T05:04:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.031+0000 7f4456a0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.031+0000 7f4456a0e640 1 -- 172.21.15.7:0/473273627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f44500ae040 con 0x7f44500a8a30 2023-11-25T05:04:40.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.082+0000 7f5f73305640 10 monclient: get_monmap_and_config 2023-11-25T05:04:40.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.082+0000 7f5f73305640 10 monclient: build_initial_monmap 2023-11-25T05:04:40.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.082+0000 7f5f73305640 10 monclient: monmap: 2023-11-25T05:04:40.085 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:40.085 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.085 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:40.083669+0000 2023-11-25T05:04:40.085 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:40.083669+0000 2023-11-25T05:04:40.085 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:40.086 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.086 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:40.086 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:40.086 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:40.086 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:40.086 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:40.086 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.083+0000 7f5f73305640 1 Processor -- start 2023-11-25T05:04:40.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.083+0000 7f5f73305640 1 -- start start 2023-11-25T05:04:40.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.083+0000 7f5f73305640 10 monclient: init 2023-11-25T05:04:40.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.084+0000 7f5f73305640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:40.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.084+0000 7f5f73305640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-11-25T05:04:40.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.084+0000 7f5f73305640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5f6c0a8a30 0x7f5f6c1424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.084+0000 7f5f73305640 10 monclient(hunting): picked mon.noname-c con 0x7f5f6c0a8a30 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:40.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.084+0000 7f5f73305640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f6c0a8e10 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:40.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.084+0000 7f5f73305640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5f6c142a20 0x7f5f6c145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.084+0000 7f5f73305640 10 monclient(hunting): picked mon.noname-a con 0x7f5f6c142a20 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:40.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.084+0000 7f5f73305640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f6c142e70 con 0x7f5f6c0a8e10 2023-11-25T05:04:40.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.084+0000 7f5f73305640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.084+0000 7f5f73305640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f6c0aaa10 con 0x7f5f6c142a20 2023-11-25T05:04:40.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.084+0000 7f5f73305640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.084+0000 7f5f73305640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f6c0aac10 con 0x7f5f6c0a8a30 2023-11-25T05:04:40.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.084+0000 7f5f73305640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:40.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.084+0000 7f5f73305640 10 monclient(hunting): authenticate will time out at 1406.326969s 2023-11-25T05:04:40.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-25T05:04:40.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.202+0000 7fcacc197640 10 monclient: get_monmap_and_config 2023-11-25T05:04:40.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.202+0000 7fcacc197640 10 monclient: build_initial_monmap 2023-11-25T05:04:40.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.202+0000 7fcacc197640 10 monclient: monmap: 2023-11-25T05:04:40.204 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:40.204 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.204 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:40.203024+0000 2023-11-25T05:04:40.204 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:40.203024+0000 2023-11-25T05:04:40.204 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:40.205 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.202+0000 7fcacc197640 1 Processor -- start 2023-11-25T05:04:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.202+0000 7fcacc197640 1 -- start start 2023-11-25T05:04:40.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.202+0000 7fcacc197640 10 monclient: init 2023-11-25T05:04:40.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.203+0000 7fcacc197640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:40.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.203+0000 7fcacc197640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-11-25T05:04:40.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.203+0000 7fcacc197640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcac40a8a30 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:40.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.203+0000 7fcacc197640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcac4142bf0 0x7fcac413ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.203+0000 7fcacc197640 10 monclient(hunting): picked mon.noname-b con 0x7fcac4142bf0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.203+0000 7fcacc197640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcac40a8e10 0x7fcac413f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.203+0000 7fcacc197640 10 monclient(hunting): picked mon.noname-c con 0x7fcac40a8e10 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.203+0000 7fcacc197640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac40aac10 con 0x7fcac40a8a30 2023-11-25T05:04:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.203+0000 7fcacc197640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.203+0000 7fcacc197640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac40aaa10 con 0x7fcac40a8e10 2023-11-25T05:04:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.203+0000 7fcacc197640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.203+0000 7fcacc197640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac40aaef0 con 0x7fcac4142bf0 2023-11-25T05:04:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.203+0000 7fcacc197640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.203+0000 7fcacc197640 10 monclient(hunting): authenticate will time out at 1406.446342s 2023-11-25T05:04:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.204+0000 7fcac9732640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcac4142bf0 0x7fcac413ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.204+0000 7fcac9732640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcac4142bf0 0x7fcac413ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41956/0 (socket says 172.21.15.7:41956) 2023-11-25T05:04:40.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.204+0000 7fcac9732640 1 -- 172.21.15.7:0/3050227440 learned_addr learned my addr 172.21.15.7:0/3050227440 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:40.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.204+0000 7fcac9732640 10 monclient(hunting): get_auth_request con 0x7fcac4142bf0 auth_method 0 2023-11-25T05:04:40.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.204+0000 7fcac9732640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:40.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.204+0000 7fcac9732640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:40.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.204+0000 7fcac9732640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:40.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.204+0000 7fcac9732640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:40.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.204+0000 7fcac9732640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:40.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.204+0000 7fcac9732640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:40.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.206+0000 7fcac9732640 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2023-11-25T05:04:40.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.206+0000 7fcac9732640 1 -- 172.21.15.7:0/3050227440 >> v2:172.21.15.90:3300/0 conn(0x7fcac40a8e10 msgr2=0x7fcac413f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:40.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.206+0000 7fcac9732640 1 --2- 172.21.15.7:0/3050227440 >> v2:172.21.15.90:3300/0 conn(0x7fcac40a8e10 0x7fcac413f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.206+0000 7fcac9732640 1 -- 172.21.15.7:0/3050227440 >> v1:172.21.15.7:6789/0 conn(0x7fcac40a8a30 legacy=0x7fcac41424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.206+0000 7fcac9732640 10 monclient: _finish_hunting 0 2023-11-25T05:04:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.206+0000 7fcac9732640 1 monclient: found mon.noname-b 2023-11-25T05:04:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.206+0000 7fcac9732640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.206+0000 7fcac9732640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:04:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.206+0000 7fcac9732640 1 -- 172.21.15.7:0/3050227440 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac40a97f0 con 0x7fcac4142bf0 2023-11-25T05:04:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.208+0000 7fcac9732640 1 --2- 172.21.15.7:0/3050227440 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcac4142bf0 0x7fcac413ecf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009870 tx=0x7fcac002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a4a915fa1f336e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.210+0000 7fcac8f31640 1 -- 172.21.15.7:0/3050227440 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0031640 con 0x7fcac4142bf0 2023-11-25T05:04:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.210+0000 7fcac8f31640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.210+0000 7fcac8f31640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.210+0000 7fcac8f31640 10 monclient: dump: 2023-11-25T05:04:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.219 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.210+0000 7fcac8f31640 10 monclient: _finish_auth 0 2023-11-25T05:04:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.210+0000 7fcac8f31640 10 monclient: _check_auth_tickets 2023-11-25T05:04:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.210+0000 7fcac8f31640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.210+0000 7fcacc197640 5 monclient: authenticate success, global_id 4319 2023-11-25T05:04:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.210+0000 7fcacc197640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.212+0000 7fcac8f31640 1 -- 172.21.15.7:0/3050227440 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac00317a0 con 0x7fcac4142bf0 2023-11-25T05:04:40.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.212+0000 7fcac8f31640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:40.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.212+0000 7fcac8f31640 1 -- 172.21.15.7:0/3050227440 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0031b30 con 0x7fcac4142bf0 2023-11-25T05:04:40.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.212+0000 7fcac8f31640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.212+0000 7fcac8f31640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:40.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.212+0000 7fcac8f31640 10 monclient: dump: 2023-11-25T05:04:40.220 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.220 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.220 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.220 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.220 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.220 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.220 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.220 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.221 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.221 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.212+0000 7fcacc197640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:40.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.212+0000 7fcacc197640 4 set_mon_vals no callback set 2023-11-25T05:04:40.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.212+0000 7fcacc197640 10 monclient: shutdown 2023-11-25T05:04:40.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.212+0000 7fcacc197640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:40.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.212+0000 7fcacc197640 1 -- 172.21.15.7:0/3050227440 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcac4142bf0 msgr2=0x7fcac413ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.212+0000 7fcacc197640 1 --2- 172.21.15.7:0/3050227440 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcac4142bf0 0x7fcac413ecf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009870 tx=0x7fcac002b8f0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.212+0000 7fcacc197640 1 -- 172.21.15.7:0/3050227440 shutdown_connections 2023-11-25T05:04:40.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.212+0000 7fcacc197640 1 --2- 172.21.15.7:0/3050227440 >> v2:172.21.15.90:3300/0 conn(0x7fcac40a8e10 0x7fcac413f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.212+0000 7fcacc197640 1 --2- 172.21.15.7:0/3050227440 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcac4142bf0 0x7fcac413ecf0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.213+0000 7fcacc197640 1 -- 172.21.15.7:0/3050227440 >> 172.21.15.7:0/3050227440 conn(0x7fcac4061a30 msgr2=0x7fcac40b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:40.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.213+0000 7fcacc197640 1 -- 172.21.15.7:0/3050227440 shutdown_connections 2023-11-25T05:04:40.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.213+0000 7fcacc197640 1 -- 172.21.15.7:0/3050227440 wait complete. 2023-11-25T05:04:40.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.213+0000 7fcacc197640 10 monclient: build_initial_monmap 2023-11-25T05:04:40.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.213+0000 7fcacc197640 10 monclient: monmap: 2023-11-25T05:04:40.222 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:40.222 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:40.222 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:40.222 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:40.222 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:40.222 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.222 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:40.222 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:40.222 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:40.222 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 1 librados: starting msgr at 2023-11-25T05:04:40.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 1 librados: starting objecter 2023-11-25T05:04:40.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 1 Processor -- start 2023-11-25T05:04:40.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 1 -- start start 2023-11-25T05:04:40.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 1 librados: setting wanted keys 2023-11-25T05:04:40.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 1 librados: calling monclient init 2023-11-25T05:04:40.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 10 monclient: init 2023-11-25T05:04:40.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:40.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-25T05:04:40.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac40a8a30 0x7fcac40aca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 10 monclient(hunting): picked mon.noname-a con 0x7fcac40a8a30 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:40.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcac40a8e10 0x7fcac40abc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 10 monclient(hunting): picked mon.noname-b con 0x7fcac40a8e10 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:40.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcac4142bf0 0x7fcac40b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 10 monclient(hunting): picked mon.noname-c con 0x7fcac4142bf0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:40.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac40ad490 con 0x7fcac40a8a30 2023-11-25T05:04:40.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac4144300 con 0x7fcac40a8e10 2023-11-25T05:04:40.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac40ad0f0 con 0x7fcac4142bf0 2023-11-25T05:04:40.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:40.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.214+0000 7fcacc197640 10 monclient(hunting): authenticate will time out at 1406.457681s 2023-11-25T05:04:40.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.215+0000 7fcaca734640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcac4142bf0 0x7fcac40b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.215+0000 7fcaca734640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcac4142bf0 0x7fcac40b08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41960/0 (socket says 172.21.15.7:41960) 2023-11-25T05:04:40.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.215+0000 7fcaca734640 1 -- 172.21.15.7:0/2127848940 learned_addr learned my addr 172.21.15.7:0/2127848940 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:40.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.215+0000 7fcaca734640 10 monclient(hunting): get_auth_request con 0x7fcac4142bf0 auth_method 0 2023-11-25T05:04:40.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.215+0000 7fcaca734640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:40.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.215+0000 7fcaca734640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:40.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.215+0000 7fcaca734640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:40.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.215+0000 7fcaca734640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:40.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.216+0000 7fcaca734640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:40.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.216+0000 7fcaca734640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:40.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.218+0000 7fcaca734640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2023-11-25T05:04:40.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.218+0000 7fcaca734640 1 -- 172.21.15.7:0/2127848940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcac40a8e10 msgr2=0x7fcac40abc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.218+0000 7fcaca734640 1 --2- 172.21.15.7:0/2127848940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcac40a8e10 0x7fcac40abc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.218+0000 7fcaca734640 1 -- 172.21.15.7:0/2127848940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac40a8a30 msgr2=0x7fcac40aca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.218+0000 7fcaca734640 1 --2- 172.21.15.7:0/2127848940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac40a8a30 0x7fcac40aca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.218+0000 7fcaca734640 10 monclient: _finish_hunting 0 2023-11-25T05:04:40.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.218+0000 7fcaca734640 1 monclient: found mon.noname-c 2023-11-25T05:04:40.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.218+0000 7fcaca734640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:40.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.218+0000 7fcaca734640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:04:40.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.218+0000 7fcaca734640 1 -- 172.21.15.7:0/2127848940 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac00094d0 con 0x7fcac4142bf0 2023-11-25T05:04:40.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.220+0000 7fcaca734640 1 --2- 172.21.15.7:0/2127848940 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcac4142bf0 0x7fcac40b08f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcabc00b9d0 tx=0x7fcabc00be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:40.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.221+0000 7fcaba7fc640 10 client.?.objecter ms_handle_connect 0x7fcac4142bf0 2023-11-25T05:04:40.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.221+0000 7fcaba7fc640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:40.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcaba7fc640 1 -- 172.21.15.7:0/2127848940 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcabc00cbd0 con 0x7fcac4142bf0 2023-11-25T05:04:40.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcaba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcaba7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:40.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcaba7fc640 10 monclient: dump: 2023-11-25T05:04:40.229 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.229 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.229 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.229 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.229 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.229 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.229 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.229 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.229 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.229 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcaba7fc640 10 monclient: _finish_auth 0 2023-11-25T05:04:40.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcaba7fc640 10 monclient: _check_auth_tickets 2023-11-25T05:04:40.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcaba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:40.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcaba7fc640 1 -- 172.21.15.7:0/2127848940 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcabc00cd50 con 0x7fcac4142bf0 2023-11-25T05:04:40.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcacc197640 5 monclient: authenticate success, global_id 4322 2023-11-25T05:04:40.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcacc197640 10 monclient: _renew_subs 2023-11-25T05:04:40.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcacc197640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:40.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcacc197640 1 -- 172.21.15.7:0/2127848940 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac4146010 con 0x7fcac4142bf0 2023-11-25T05:04:40.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcaba7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:40.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcaba7fc640 1 -- 172.21.15.7:0/2127848940 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcabc014680 con 0x7fcac4142bf0 2023-11-25T05:04:40.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcaba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcaba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:40.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcaba7fc640 10 monclient: dump: 2023-11-25T05:04:40.230 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.231 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.231 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.231 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.231 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.231 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.231 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.231 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.231 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.231 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcabbfff640 4 set_mon_vals no callback set 2023-11-25T05:04:40.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcacc197640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:40.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcacc197640 10 monclient: _renew_subs 2023-11-25T05:04:40.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcacc197640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:40.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcacc197640 1 -- 172.21.15.7:0/2127848940 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac40b0e30 con 0x7fcac4142bf0 2023-11-25T05:04:40.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.224+0000 7fcacc197640 1 librados: init done 2023-11-25T05:04:40.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.225+0000 7fcacc197640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:40.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.225+0000 7fcacc197640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:40.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.225+0000 7fcacc197640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:40.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.225+0000 7fcacc197640 1 -- 172.21.15.7:0/2127848940 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac4067ac0 con 0x7fcac4142bf0 2023-11-25T05:04:40.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.246+0000 7fcaba7fc640 1 -- 172.21.15.7:0/2127848940 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcabc01d3b0 con 0x7fcac4142bf0 2023-11-25T05:04:40.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.246+0000 7fcaba7fc640 1 --2- 172.21.15.7:0/2127848940 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fcaa806c820 0x7fcaa806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.246+0000 7fcac9f33640 1 --2- 172.21.15.7:0/2127848940 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fcaa806c820 0x7fcaa806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.246+0000 7fcac9f33640 10 monclient: get_auth_request con 0x7fcaa806c820 auth_method 0 2023-11-25T05:04:40.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.247+0000 7fcac9f33640 1 --2- 172.21.15.7:0/2127848940 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fcaa806c820 0x7fcaa806ece0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcab4001560 tx=0x7fcab4007990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:40.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.247+0000 7fcaba7fc640 10 client.4322.objecter ms_handle_connect 0x7fcaa806c820 2023-11-25T05:04:40.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.260+0000 7f6a5e7fc640 1 -- 172.21.15.7:0/3478383938 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6a60040d00 con 0x7f6a680ac070 2023-11-25T05:04:40.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.260+0000 7f6a5e7fc640 1 --2- 172.21.15.7:0/3478383938 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6a4c06c850 0x7f6a4c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.261+0000 7f6a6dbed640 1 --2- 172.21.15.7:0/3478383938 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6a4c06c850 0x7f6a4c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.261+0000 7f6a6dbed640 10 monclient: get_auth_request con 0x7f6a4c06c850 auth_method 0 2023-11-25T05:04:40.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.261+0000 7f6a6dbed640 1 --2- 172.21.15.7:0/3478383938 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6a4c06c850 0x7f6a4c06ed10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a640015d0 tx=0x7f6a64009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:40.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.261+0000 7f6a5e7fc640 10 client.4378.objecter ms_handle_connect 0x7f6a4c06c850 2023-11-25T05:04:40.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.263+0000 7f6a5e7fc640 1 -- 172.21.15.7:0/3478383938 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6a600094f0 con 0x7f6a680ac070 2023-11-25T05:04:40.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.263+0000 7f6a5e7fc640 10 client.4378.objecter ms_dispatch 0x7f6a68000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:40.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.263+0000 7f6a5e7fc640 3 client.4378.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:40.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.263+0000 7f6a5e7fc640 3 client.4378.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:40.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.263+0000 7f6a5e7fc640 20 client.4378.objecter dump_active .. 0 homeless 2023-11-25T05:04:40.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.268+0000 7fdf037fe640 1 -- 172.21.15.7:0/1947578733 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf0c054890 con 0x7fdf1c0a8a30 2023-11-25T05:04:40.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.268+0000 7fdf037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:40.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.268+0000 7fdf037fe640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:40.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.273+0000 7f6a5e7fc640 1 -- 172.21.15.7:0/3478383938 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a60077d50 con 0x7f6a680ac070 2023-11-25T05:04:40.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.273+0000 7f6a5e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:40.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.273+0000 7f6a5e7fc640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:40.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.273+0000 7f5f710a1640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5f6c0a8a30 0x7f5f6c1424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.274+0000 7f5f710a1640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5f6c0a8a30 0x7f5f6c1424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:35296/0 (socket says 172.21.15.7:35296) 2023-11-25T05:04:40.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.274+0000 7f5f710a1640 1 -- 172.21.15.7:0/1104052078 learned_addr learned my addr 172.21.15.7:0/1104052078 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:40.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.274+0000 7f5f710a1640 10 monclient(hunting): get_auth_request con 0x7f5f6c0a8a30 auth_method 0 2023-11-25T05:04:40.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.274+0000 7f5f710a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:40.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.274+0000 7f5f710a1640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:40.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.274+0000 7f5f710a1640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:40.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.276+0000 7f5f710a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:40.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.276+0000 7f5f710a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:40.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.276+0000 7f5f710a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.278+0000 7f5f710a1640 10 monclient(hunting): handle_auth_done global_id 4381 payload 274 2023-11-25T05:04:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.278+0000 7f5f710a1640 1 -- 172.21.15.7:0/1104052078 >> v2:172.21.15.7:3300/0 conn(0x7f5f6c142a20 msgr2=0x7f5f6c145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.278+0000 7f5f710a1640 1 --2- 172.21.15.7:0/1104052078 >> v2:172.21.15.7:3300/0 conn(0x7f5f6c142a20 0x7f5f6c145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.278+0000 7f5f710a1640 1 -- 172.21.15.7:0/1104052078 >> v1:172.21.15.90:6789/0 conn(0x7f5f6c0a8e10 legacy=0x7f5f6c1454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.278+0000 7f5f710a1640 10 monclient: _finish_hunting 0 2023-11-25T05:04:40.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.278+0000 7f5f710a1640 1 monclient: found mon.noname-c 2023-11-25T05:04:40.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.278+0000 7f5f710a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:40.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.278+0000 7f5f710a1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:40.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.278+0000 7f5f710a1640 1 -- 172.21.15.7:0/1104052078 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6c0aaef0 con 0x7f5f6c0a8a30 2023-11-25T05:04:40.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.282+0000 7f5f710a1640 1 --2- 172.21.15.7:0/1104052078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f6c0a8a30 0x7f5f6c1424e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c0b5470 tx=0x7f5f6802ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1817b57098b35e62 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:40.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f63fff640 1 -- 172.21.15.7:0/1104052078 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6802f850 con 0x7f5f6c0a8a30 2023-11-25T05:04:40.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f63fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:40.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f63fff640 10 monclient: dump: 2023-11-25T05:04:40.290 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.290 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.290 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.290 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.290 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.290 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.290 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.291 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.291 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.291 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f63fff640 10 monclient: _finish_auth 0 2023-11-25T05:04:40.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f63fff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:40.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:40.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f63fff640 1 -- 172.21.15.7:0/1104052078 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6802f9b0 con 0x7f5f6c0a8a30 2023-11-25T05:04:40.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f63fff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:40.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f63fff640 1 -- 172.21.15.7:0/1104052078 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6802fd40 con 0x7f5f6c0a8a30 2023-11-25T05:04:40.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:40.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f63fff640 10 monclient: dump: 2023-11-25T05:04:40.291 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.292 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.292 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.292 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.292 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.292 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.292 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.292 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.292 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.292 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f73305640 5 monclient: authenticate success, global_id 4381 2023-11-25T05:04:40.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f73305640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:40.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.286+0000 7f5f73305640 4 set_mon_vals no callback set 2023-11-25T05:04:40.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.287+0000 7f5f73305640 10 monclient: shutdown 2023-11-25T05:04:40.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.287+0000 7f5f73305640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:40.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.287+0000 7f5f73305640 1 -- 172.21.15.7:0/1104052078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f6c0a8a30 msgr2=0x7f5f6c1424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.287+0000 7f5f73305640 1 --2- 172.21.15.7:0/1104052078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f6c0a8a30 0x7f5f6c1424e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c0b5470 tx=0x7f5f6802ba30 comp rx=0 tx=0).stop 2023-11-25T05:04:40.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.287+0000 7f5f73305640 1 -- 172.21.15.7:0/1104052078 shutdown_connections 2023-11-25T05:04:40.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.287+0000 7f5f73305640 1 --2- 172.21.15.7:0/1104052078 >> v2:172.21.15.7:3300/0 conn(0x7f5f6c142a20 0x7f5f6c145bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.287+0000 7f5f73305640 1 --2- 172.21.15.7:0/1104052078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f6c0a8a30 0x7f5f6c1424e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.287+0000 7f5f73305640 1 -- 172.21.15.7:0/1104052078 >> 172.21.15.7:0/1104052078 conn(0x7f5f6c061a30 msgr2=0x7f5f6c0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:40.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.287+0000 7f5f73305640 1 -- 172.21.15.7:0/1104052078 shutdown_connections 2023-11-25T05:04:40.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.287+0000 7f5f73305640 1 -- 172.21.15.7:0/1104052078 wait complete. 2023-11-25T05:04:40.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.288+0000 7f5f73305640 10 monclient: build_initial_monmap 2023-11-25T05:04:40.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.288+0000 7f5f73305640 10 monclient: monmap: 2023-11-25T05:04:40.293 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:40.293 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:40.293 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:40.294 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:40.294 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:40.294 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.294 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:40.294 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:40.294 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:40.294 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.288+0000 7f5f73305640 1 librados: starting msgr at 2023-11-25T05:04:40.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.288+0000 7f5f73305640 1 librados: starting objecter 2023-11-25T05:04:40.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.288+0000 7f5f73305640 1 Processor -- start 2023-11-25T05:04:40.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.289+0000 7f5f73305640 1 -- start start 2023-11-25T05:04:40.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.289+0000 7f5f73305640 1 librados: setting wanted keys 2023-11-25T05:04:40.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.289+0000 7f5f73305640 1 librados: calling monclient init 2023-11-25T05:04:40.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.289+0000 7f5f73305640 10 monclient: init 2023-11-25T05:04:40.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.289+0000 7f5f73305640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:40.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.289+0000 7f5f73305640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:04:40.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.289+0000 7f5f73305640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f6c0a8a30 0x7f5f6c0aed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.290+0000 7f5f73305640 10 monclient(hunting): picked mon.noname-a con 0x7f5f6c0a8a30 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:40.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.290+0000 7f5f73305640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f6c0a8e10 0x7f5f6c0af930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.290+0000 7f5f73305640 10 monclient(hunting): picked mon.noname-c con 0x7f5f6c0a8e10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:40.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.290+0000 7f5f73305640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f6c142a20 0x7f5f6c1da7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.290+0000 7f5f73305640 10 monclient(hunting): picked mon.noname-b con 0x7f5f6c142a20 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:40.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.290+0000 7f5f73305640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.290+0000 7f5f73305640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c1da3a0 con 0x7f5f6c0a8a30 2023-11-25T05:04:40.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.290+0000 7f5f73305640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.290+0000 7f5f73305640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c0ac420 con 0x7f5f6c142a20 2023-11-25T05:04:40.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.290+0000 7f5f73305640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.290+0000 7f5f73305640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c0afe70 con 0x7f5f6c0a8e10 2023-11-25T05:04:40.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.290+0000 7f5f73305640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:40.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.290+0000 7f5f73305640 10 monclient(hunting): authenticate will time out at 1406.533136s 2023-11-25T05:04:40.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.290+0000 7f5f718a2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f6c142a20 0x7f5f6c1da7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.291+0000 7f5f718a2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f6c142a20 0x7f5f6c1da7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:35330/0 (socket says 172.21.15.7:35330) 2023-11-25T05:04:40.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.291+0000 7f5f718a2640 1 -- 172.21.15.7:0/3304572429 learned_addr learned my addr 172.21.15.7:0/3304572429 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:40.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.291+0000 7f5f718a2640 10 monclient(hunting): get_auth_request con 0x7f5f6c142a20 auth_method 0 2023-11-25T05:04:40.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.291+0000 7f5f718a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:40.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.291+0000 7f5f718a2640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:40.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.291+0000 7f5f718a2640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:40.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.291+0000 7f5f718a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:40.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.291+0000 7f5f718a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:40.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.291+0000 7f5f718a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:40.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.294+0000 7f5f718a2640 10 monclient(hunting): handle_auth_done global_id 4384 payload 824 2023-11-25T05:04:40.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.294+0000 7f5f718a2640 1 -- 172.21.15.7:0/3304572429 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f6c0a8e10 msgr2=0x7f5f6c0af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.294+0000 7f5f718a2640 1 --2- 172.21.15.7:0/3304572429 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f6c0a8e10 0x7f5f6c0af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.294+0000 7f5f718a2640 1 -- 172.21.15.7:0/3304572429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f6c0a8a30 msgr2=0x7f5f6c0aed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.294+0000 7f5f718a2640 1 --2- 172.21.15.7:0/3304572429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f6c0a8a30 0x7f5f6c0aed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.294+0000 7f5f718a2640 10 monclient: _finish_hunting 0 2023-11-25T05:04:40.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.294+0000 7f5f718a2640 1 monclient: found mon.noname-b 2023-11-25T05:04:40.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.294+0000 7f5f718a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:40.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.294+0000 7f5f718a2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:40.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.294+0000 7f5f718a2640 1 -- 172.21.15.7:0/3304572429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f680094d0 con 0x7f5f6c142a20 2023-11-25T05:04:40.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.296+0000 7f5f718a2640 1 --2- 172.21.15.7:0/3304572429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f6c142a20 0x7f5f6c1da7f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5f6400b970 tx=0x7f5f6400be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:40.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.296+0000 7f5f61ffb640 10 client.?.objecter ms_handle_connect 0x7f5f6c142a20 2023-11-25T05:04:40.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.296+0000 7f5f61ffb640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:40.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f61ffb640 1 -- 172.21.15.7:0/3304572429 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6400cb40 con 0x7f5f6c142a20 2023-11-25T05:04:40.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f61ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:40.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f61ffb640 10 monclient: dump: 2023-11-25T05:04:40.341 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.341 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.341 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.341 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.341 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.341 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.341 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.341 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.342 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.342 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f61ffb640 10 monclient: _finish_auth 0 2023-11-25T05:04:40.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f61ffb640 10 monclient: _check_auth_tickets 2023-11-25T05:04:40.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:40.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f61ffb640 1 -- 172.21.15.7:0/3304572429 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6400ccc0 con 0x7f5f6c142a20 2023-11-25T05:04:40.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f73305640 5 monclient: authenticate success, global_id 4384 2023-11-25T05:04:40.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f73305640 10 monclient: _renew_subs 2023-11-25T05:04:40.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f73305640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:40.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f73305640 1 -- 172.21.15.7:0/3304572429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6c0b15d0 con 0x7f5f6c142a20 2023-11-25T05:04:40.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f61ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:40.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f61ffb640 1 -- 172.21.15.7:0/3304572429 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f64014680 con 0x7f5f6c142a20 2023-11-25T05:04:40.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f61ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:40.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f61ffb640 10 monclient: dump: 2023-11-25T05:04:40.343 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.343 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.343 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.343 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.343 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.343 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.343 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.343 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.343 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.343 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f637fe640 4 set_mon_vals no callback set 2023-11-25T05:04:40.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f73305640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:40.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f73305640 10 monclient: _renew_subs 2023-11-25T05:04:40.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f73305640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:40.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f73305640 1 -- 172.21.15.7:0/3304572429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6c1daf10 con 0x7f5f6c142a20 2023-11-25T05:04:40.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.300+0000 7f5f73305640 1 librados: init done 2023-11-25T05:04:40.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.301+0000 7f5f73305640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:40.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.301+0000 7f5f73305640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:40.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.301+0000 7f5f73305640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:40.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.301+0000 7f5f73305640 1 -- 172.21.15.7:0/3304572429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6c067ac0 con 0x7f5f6c142a20 2023-11-25T05:04:40.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.319+0000 7f5f61ffb640 1 -- 172.21.15.7:0/3304572429 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5f6401d3b0 con 0x7f5f6c142a20 2023-11-25T05:04:40.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.319+0000 7f5f61ffb640 1 --2- 172.21.15.7:0/3304572429 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f5f5006c820 0x7f5f5006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.319+0000 7f5f710a1640 1 --2- 172.21.15.7:0/3304572429 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f5f5006c820 0x7f5f5006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.319+0000 7f5f710a1640 10 monclient: get_auth_request con 0x7f5f5006c820 auth_method 0 2023-11-25T05:04:40.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.320+0000 7f5f710a1640 1 --2- 172.21.15.7:0/3304572429 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f5f5006c820 0x7f5f5006ece0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5f68036000 tx=0x7f5f68030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:40.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.320+0000 7f5f61ffb640 10 client.4384.objecter ms_handle_connect 0x7f5f5006c820 2023-11-25T05:04:40.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.322+0000 7f5f61ffb640 1 -- 172.21.15.7:0/3304572429 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5f6400ce30 con 0x7f5f6c142a20 2023-11-25T05:04:40.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.322+0000 7f5f61ffb640 10 client.4384.objecter ms_dispatch 0x7f5f6c000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:40.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.322+0000 7f5f61ffb640 3 client.4384.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:40.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.322+0000 7f5f61ffb640 3 client.4384.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:40.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.322+0000 7f5f61ffb640 20 client.4384.objecter dump_active .. 0 homeless 2023-11-25T05:04:40.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.339+0000 7f444cff9640 1 -- 172.21.15.7:0/473273627 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f44500ae040 con 0x7f44500a8a30 2023-11-25T05:04:40.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.339+0000 7f444cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-11-25T05:04:40.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.339+0000 7f444cff9640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:40.346 INFO:teuthology.orchestra.run.smithi007.stdout:55834574853 2023-11-25T05:04:40.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 10 librados: watch_flush enter 2023-11-25T05:04:40.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 10 librados: watch_flush exit 2023-11-25T05:04:40.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 20 client.4353.objecter shutdown clearing up homeless session... 2023-11-25T05:04:40.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 10 client.4353.objecter successfully canceled tick 2023-11-25T05:04:40.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 1 -- 172.21.15.7:0/473273627 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f443c06c820 msgr2=0x7f443c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 1 --2- 172.21.15.7:0/473273627 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f443c06c820 0x7f443c06ece0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4444036000 tx=0x7f4444030000 comp rx=0 tx=0).stop 2023-11-25T05:04:40.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 10 monclient: shutdown 2023-11-25T05:04:40.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:40.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 1 -- 172.21.15.7:0/473273627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44500a8a30 msgr2=0x7f4450138a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 1 --2- 172.21.15.7:0/473273627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44500a8a30 0x7f4450138a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f444000c400 tx=0x7f444000c8c0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 1 -- 172.21.15.7:0/473273627 shutdown_connections 2023-11-25T05:04:40.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 1 --2- 172.21.15.7:0/473273627 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f443c06c820 0x7f443c06ece0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 1 --2- 172.21.15.7:0/473273627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4450142bf0 0x7f44500ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 1 --2- 172.21.15.7:0/473273627 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44500a8e10 0x7f4450138f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 1 --2- 172.21.15.7:0/473273627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44500a8a30 0x7f4450138a20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.342+0000 7f4456a0e640 1 -- 172.21.15.7:0/473273627 >> 172.21.15.7:0/473273627 conn(0x7f4450061a30 msgr2=0x7f44501424e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:40.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.343+0000 7f4456a0e640 1 -- 172.21.15.7:0/473273627 shutdown_connections 2023-11-25T05:04:40.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.344+0000 7f4456a0e640 1 -- 172.21.15.7:0/473273627 wait complete. 2023-11-25T05:04:40.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.344+0000 7f4456a0e640 1 librados: shutdown 2023-11-25T05:04:40.409 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.7 2023-11-25T05:04:40.409 DEBUG:teuthology.parallel:result is None 2023-11-25T05:04:40.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.471+0000 7fdf212af640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-11-25T05:04:40.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.471+0000 7fdf212af640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-11-25T05:04:40.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.471+0000 7fdf212af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:40.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.471+0000 7fdf212af640 1 -- 172.21.15.7:0/1947578733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fdf1c0aded0 con 0x7fdf1c0a8a30 2023-11-25T05:04:40.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.476+0000 7fdf037fe640 1 -- 172.21.15.7:0/1947578733 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdf1c0aded0 con 0x7fdf1c0a8a30 2023-11-25T05:04:40.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.476+0000 7fdf037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-11-25T05:04:40.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.476+0000 7fdf037fe640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:40.478 INFO:teuthology.orchestra.run.smithi007.stdout:68719476742 2023-11-25T05:04:40.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.477+0000 7f6a6fe51640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-11-25T05:04:40.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.477+0000 7f6a6fe51640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-11-25T05:04:40.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.477+0000 7f6a6fe51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:40.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.477+0000 7f6a6fe51640 1 -- 172.21.15.7:0/3478383938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6a680abc70 con 0x7f6a680ac070 2023-11-25T05:04:40.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 10 librados: watch_flush enter 2023-11-25T05:04:40.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 10 librados: watch_flush exit 2023-11-25T05:04:40.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 20 client.4356.objecter shutdown clearing up homeless session... 2023-11-25T05:04:40.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 10 client.4356.objecter successfully canceled tick 2023-11-25T05:04:40.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 1 -- 172.21.15.7:0/1947578733 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdf0806c7d0 msgr2=0x7fdf0806ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 1 --2- 172.21.15.7:0/1947578733 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdf0806c7d0 0x7fdf0806ec90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdf10006a90 tx=0x7fdf10006a20 comp rx=0 tx=0).stop 2023-11-25T05:04:40.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 10 monclient: shutdown 2023-11-25T05:04:40.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:40.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 1 -- 172.21.15.7:0/1947578733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf1c0a8a30 msgr2=0x7fdf1c0abf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 1 --2- 172.21.15.7:0/1947578733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf1c0a8a30 0x7fdf1c0abf10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c00c450 tx=0x7fdf0c00c910 comp rx=0 tx=0).stop 2023-11-25T05:04:40.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 1 -- 172.21.15.7:0/1947578733 shutdown_connections 2023-11-25T05:04:40.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 1 --2- 172.21.15.7:0/1947578733 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdf0806c7d0 0x7fdf0806ec90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 1 --2- 172.21.15.7:0/1947578733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf1c142bf0 0x7fdf1c144580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 1 --2- 172.21.15.7:0/1947578733 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdf1c0a8e10 0x7fdf1c0ac450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 1 --2- 172.21.15.7:0/1947578733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf1c0a8a30 0x7fdf1c0abf10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 1 -- 172.21.15.7:0/1947578733 >> 172.21.15.7:0/1947578733 conn(0x7fdf1c061a30 msgr2=0x7fdf1c0b25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:40.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 1 -- 172.21.15.7:0/1947578733 shutdown_connections 2023-11-25T05:04:40.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 1 -- 172.21.15.7:0/1947578733 wait complete. 2023-11-25T05:04:40.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.478+0000 7fdf212af640 1 librados: shutdown 2023-11-25T05:04:40.491 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476742 for osd.1 2023-11-25T05:04:40.491 DEBUG:teuthology.parallel:result is None 2023-11-25T05:04:40.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.519+0000 7fcaba7fc640 1 -- 172.21.15.7:0/2127848940 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcabc058ea0 con 0x7fcac4142bf0 2023-11-25T05:04:40.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.519+0000 7fcaba7fc640 10 client.4322.objecter ms_dispatch 0x7fcac4000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:40.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.519+0000 7fcaba7fc640 3 client.4322.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:40.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.519+0000 7fcaba7fc640 3 client.4322.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:40.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.519+0000 7fcaba7fc640 20 client.4322.objecter dump_active .. 0 homeless 2023-11-25T05:04:40.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.520+0000 7fcaba7fc640 1 -- 172.21.15.7:0/2127848940 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcabc054ae0 con 0x7fcac4142bf0 2023-11-25T05:04:40.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.520+0000 7fcaba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:40.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.520+0000 7fcaba7fc640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:40.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.558+0000 7f5f61ffb640 1 -- 172.21.15.7:0/3304572429 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f64054ac0 con 0x7f5f6c142a20 2023-11-25T05:04:40.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.558+0000 7f5f61ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:40.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.558+0000 7f5f61ffb640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:40.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.566+0000 7f6a5e7fc640 1 -- 172.21.15.7:0/3478383938 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6a6007b370 con 0x7f6a680ac070 2023-11-25T05:04:40.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.566+0000 7f6a5e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-11-25T05:04:40.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.566+0000 7f6a5e7fc640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:40.568 INFO:teuthology.orchestra.run.smithi007.stdout:55834574853 2023-11-25T05:04:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 10 librados: watch_flush enter 2023-11-25T05:04:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 10 librados: watch_flush exit 2023-11-25T05:04:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 20 client.4378.objecter shutdown clearing up homeless session... 2023-11-25T05:04:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 10 client.4378.objecter successfully canceled tick 2023-11-25T05:04:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 1 -- 172.21.15.7:0/3478383938 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6a4c06c850 msgr2=0x7f6a4c06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 1 --2- 172.21.15.7:0/3478383938 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6a4c06c850 0x7f6a4c06ed10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a640015d0 tx=0x7f6a64009040 comp rx=0 tx=0).stop 2023-11-25T05:04:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 10 monclient: shutdown 2023-11-25T05:04:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 1 -- 172.21.15.7:0/3478383938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a680ac070 msgr2=0x7f6a68139480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 1 --2- 172.21.15.7:0/3478383938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a680ac070 0x7f6a68139480 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6a60032000 tx=0x7f6a60004a40 comp rx=0 tx=0).stop 2023-11-25T05:04:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 1 -- 172.21.15.7:0/3478383938 shutdown_connections 2023-11-25T05:04:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 1 --2- 172.21.15.7:0/3478383938 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6a4c06c850 0x7f6a4c06ed10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 1 --2- 172.21.15.7:0/3478383938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a680ac070 0x7f6a68139480 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 1 --2- 172.21.15.7:0/3478383938 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a68141f90 0x7f6a68138f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 1 --2- 172.21.15.7:0/3478383938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a680a8910 0x7f6a680ab540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 1 -- 172.21.15.7:0/3478383938 >> 172.21.15.7:0/3478383938 conn(0x7f6a680b7780 msgr2=0x7f6a68138230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 1 -- 172.21.15.7:0/3478383938 shutdown_connections 2023-11-25T05:04:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 1 -- 172.21.15.7:0/3478383938 wait complete. 2023-11-25T05:04:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.568+0000 7f6a6fe51640 1 librados: shutdown 2023-11-25T05:04:40.581 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.5 2023-11-25T05:04:40.581 DEBUG:teuthology.parallel:result is None 2023-11-25T05:04:40.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-25T05:04:40.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.704+0000 7fcacc197640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-11-25T05:04:40.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.704+0000 7fcacc197640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-11-25T05:04:40.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.704+0000 7fcacc197640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:40.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.704+0000 7fcacc197640 1 -- 172.21.15.7:0/2127848940 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fcac41da630 con 0x7fcac4142bf0 2023-11-25T05:04:40.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.708+0000 7fcaba7fc640 1 -- 172.21.15.7:0/2127848940 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcabc019020 con 0x7fcac4142bf0 2023-11-25T05:04:40.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.708+0000 7fcaba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-11-25T05:04:40.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.708+0000 7fcaba7fc640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:40.711 INFO:teuthology.orchestra.run.smithi007.stdout:64424509447 2023-11-25T05:04:40.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 10 librados: watch_flush enter 2023-11-25T05:04:40.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 10 librados: watch_flush exit 2023-11-25T05:04:40.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 20 client.4322.objecter shutdown clearing up homeless session... 2023-11-25T05:04:40.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 10 client.4322.objecter successfully canceled tick 2023-11-25T05:04:40.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 1 -- 172.21.15.7:0/2127848940 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fcaa806c820 msgr2=0x7fcaa806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 1 --2- 172.21.15.7:0/2127848940 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fcaa806c820 0x7fcaa806ece0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcab4001560 tx=0x7fcab4007990 comp rx=0 tx=0).stop 2023-11-25T05:04:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 10 monclient: shutdown 2023-11-25T05:04:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 1 -- 172.21.15.7:0/2127848940 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcac4142bf0 msgr2=0x7fcac40b08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 1 --2- 172.21.15.7:0/2127848940 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcac4142bf0 0x7fcac40b08f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcabc00b9d0 tx=0x7fcabc00be90 comp rx=0 tx=0).stop 2023-11-25T05:04:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 1 -- 172.21.15.7:0/2127848940 shutdown_connections 2023-11-25T05:04:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 1 --2- 172.21.15.7:0/2127848940 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fcaa806c820 0x7fcaa806ece0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 1 --2- 172.21.15.7:0/2127848940 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcac4142bf0 0x7fcac40b08f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 1 --2- 172.21.15.7:0/2127848940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcac40a8e10 0x7fcac40abc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 1 --2- 172.21.15.7:0/2127848940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac40a8a30 0x7fcac40aca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 1 -- 172.21.15.7:0/2127848940 >> 172.21.15.7:0/2127848940 conn(0x7fcac4061a30 msgr2=0x7fcac413cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 1 -- 172.21.15.7:0/2127848940 shutdown_connections 2023-11-25T05:04:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.710+0000 7fcacc197640 1 -- 172.21.15.7:0/2127848940 wait complete. 2023-11-25T05:04:40.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.711+0000 7fcacc197640 1 librados: shutdown 2023-11-25T05:04:40.726 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.4 2023-11-25T05:04:40.726 DEBUG:teuthology.parallel:result is None 2023-11-25T05:04:40.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-25T05:04:40.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.745+0000 7f5f73305640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-11-25T05:04:40.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.745+0000 7f5f73305640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-11-25T05:04:40.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.745+0000 7f5f73305640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:40.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.745+0000 7f5f73305640 1 -- 172.21.15.7:0/3304572429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f5f6c1da510 con 0x7f5f6c142a20 2023-11-25T05:04:40.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.749+0000 7f5f61ffb640 1 -- 172.21.15.7:0/3304572429 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5f6c1da510 con 0x7f5f6c142a20 2023-11-25T05:04:40.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.749+0000 7f5f61ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-11-25T05:04:40.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.749+0000 7f5f61ffb640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:40.751 INFO:teuthology.orchestra.run.smithi007.stdout:68719476741 2023-11-25T05:04:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 10 librados: watch_flush enter 2023-11-25T05:04:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 10 librados: watch_flush exit 2023-11-25T05:04:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 20 client.4384.objecter shutdown clearing up homeless session... 2023-11-25T05:04:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 10 client.4384.objecter successfully canceled tick 2023-11-25T05:04:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 1 -- 172.21.15.7:0/3304572429 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f5f5006c820 msgr2=0x7f5f5006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 1 --2- 172.21.15.7:0/3304572429 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f5f5006c820 0x7f5f5006ece0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5f68036000 tx=0x7f5f68030000 comp rx=0 tx=0).stop 2023-11-25T05:04:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 10 monclient: shutdown 2023-11-25T05:04:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 1 -- 172.21.15.7:0/3304572429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f6c142a20 msgr2=0x7f5f6c1da7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 1 --2- 172.21.15.7:0/3304572429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f6c142a20 0x7f5f6c1da7f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5f6400b970 tx=0x7f5f6400be30 comp rx=0 tx=0).stop 2023-11-25T05:04:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 1 -- 172.21.15.7:0/3304572429 shutdown_connections 2023-11-25T05:04:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 1 --2- 172.21.15.7:0/3304572429 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f5f5006c820 0x7f5f5006ece0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 1 --2- 172.21.15.7:0/3304572429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f6c142a20 0x7f5f6c1da7f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 1 --2- 172.21.15.7:0/3304572429 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f6c0a8e10 0x7f5f6c0af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 1 --2- 172.21.15.7:0/3304572429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f6c0a8a30 0x7f5f6c0aed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 1 -- 172.21.15.7:0/3304572429 >> 172.21.15.7:0/3304572429 conn(0x7f5f6c061a30 msgr2=0x7f5f6c0b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 1 -- 172.21.15.7:0/3304572429 shutdown_connections 2023-11-25T05:04:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 1 -- 172.21.15.7:0/3304572429 wait complete. 2023-11-25T05:04:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.751+0000 7f5f73305640 1 librados: shutdown 2023-11-25T05:04:40.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.752+0000 7f0a9a83f640 10 monclient: get_monmap_and_config 2023-11-25T05:04:40.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.752+0000 7f0a9a83f640 10 monclient: build_initial_monmap 2023-11-25T05:04:40.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.752+0000 7f0a9a83f640 10 monclient: monmap: 2023-11-25T05:04:40.757 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:40.753545+0000 2023-11-25T05:04:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:40.753545+0000 2023-11-25T05:04:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:40.758 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.753+0000 7f0a9a83f640 1 Processor -- start 2023-11-25T05:04:40.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.753+0000 7f0a9a83f640 1 -- start start 2023-11-25T05:04:40.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.753+0000 7f0a9a83f640 10 monclient: init 2023-11-25T05:04:40.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.754+0000 7f0a9a83f640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:40.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.754+0000 7f0a9a83f640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-11-25T05:04:40.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.754+0000 7f0a9a83f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a940a8a30 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:40.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.754+0000 7f0a9a83f640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0a94142bf0 0x7f0a9413ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.754+0000 7f0a9a83f640 10 monclient(hunting): picked mon.noname-c con 0x7f0a94142bf0 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:40.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.754+0000 7f0a9a83f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0a940a8e10 0x7f0a9413f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.754+0000 7f0a9a83f640 10 monclient(hunting): picked mon.noname-a con 0x7f0a940a8e10 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:40.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.754+0000 7f0a9a83f640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a940aac10 con 0x7f0a940a8a30 2023-11-25T05:04:40.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.754+0000 7f0a9a83f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.754+0000 7f0a9a83f640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a940aaa10 con 0x7f0a940a8e10 2023-11-25T05:04:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.754+0000 7f0a9a83f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.754+0000 7f0a9a83f640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a940aaef0 con 0x7f0a94142bf0 2023-11-25T05:04:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.754+0000 7f0a9a83f640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.754+0000 7f0a9a83f640 10 monclient(hunting): authenticate will time out at 1406.997244s 2023-11-25T05:04:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.755+0000 7f0a937fe640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0a94142bf0 0x7f0a9413ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.755+0000 7f0a937fe640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0a94142bf0 0x7f0a9413ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:35334/0 (socket says 172.21.15.7:35334) 2023-11-25T05:04:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.755+0000 7f0a937fe640 1 -- 172.21.15.7:0/2105173744 learned_addr learned my addr 172.21.15.7:0/2105173744 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.755+0000 7f0a937fe640 10 monclient(hunting): get_auth_request con 0x7f0a94142bf0 auth_method 0 2023-11-25T05:04:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.755+0000 7f0a937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.755+0000 7f0a937fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.755+0000 7f0a937fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.756+0000 7f0a937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.756+0000 7f0a937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.756+0000 7f0a937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:40.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.756+0000 7f0a98ddc640 1 --2- 172.21.15.7:0/2105173744 >> v2:172.21.15.7:3300/0 conn(0x7f0a940a8e10 0x7f0a9413f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a98ddc640 10 monclient(hunting): get_auth_request con 0x7f0a940a8e10 auth_method 0 2023-11-25T05:04:40.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a98ddc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:40.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a98ddc640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:40.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a98ddc640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:40.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a98ddc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:40.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a98ddc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:40.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a98ddc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:40.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a937fe640 10 monclient(hunting): handle_auth_done global_id 4387 payload 274 2023-11-25T05:04:40.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a937fe640 1 -- 172.21.15.7:0/2105173744 >> v2:172.21.15.7:3300/0 conn(0x7f0a940a8e10 msgr2=0x7f0a9413f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:40.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a937fe640 1 --2- 172.21.15.7:0/2105173744 >> v2:172.21.15.7:3300/0 conn(0x7f0a940a8e10 0x7f0a9413f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a937fe640 1 -- 172.21.15.7:0/2105173744 >> v1:172.21.15.90:6789/0 conn(0x7f0a940a8a30 legacy=0x7f0a941424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:40.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a937fe640 10 monclient: _finish_hunting 0 2023-11-25T05:04:40.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a937fe640 1 monclient: found mon.noname-c 2023-11-25T05:04:40.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:40.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a937fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:40.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.757+0000 7f0a937fe640 1 -- 172.21.15.7:0/2105173744 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a940a97f0 con 0x7f0a94142bf0 2023-11-25T05:04:40.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.760+0000 7f0a937fe640 1 --2- 172.21.15.7:0/2105173744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a94142bf0 0x7f0a9413ecf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0a940b87d0 tx=0x7f0a8802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bf7eff7157bcee server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:40.765 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476741 for osd.3 2023-11-25T05:04:40.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a92ffd640 1 -- 172.21.15.7:0/2105173744 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a88030860 con 0x7f0a94142bf0 2023-11-25T05:04:40.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a92ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:40.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a92ffd640 10 monclient: dump: 2023-11-25T05:04:40.766 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.766 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.766 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.766 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.766 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.766 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.766 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.767 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.767 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.767 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a92ffd640 10 monclient: _finish_auth 0 2023-11-25T05:04:40.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a92ffd640 10 monclient: _check_auth_tickets 2023-11-25T05:04:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a9a83f640 5 monclient: authenticate success, global_id 4387 2023-11-25T05:04:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a92ffd640 1 -- 172.21.15.7:0/2105173744 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a880309c0 con 0x7f0a94142bf0 2023-11-25T05:04:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a92ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a9a83f640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a9a83f640 4 set_mon_vals no callback set 2023-11-25T05:04:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a92ffd640 1 -- 172.21.15.7:0/2105173744 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a88030d50 con 0x7f0a94142bf0 2023-11-25T05:04:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a92ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.764+0000 7f0a92ffd640 10 monclient: dump: 2023-11-25T05:04:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.769 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.769 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.769 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.769 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.769 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.769 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.769 DEBUG:teuthology.parallel:result is None 2023-11-25T05:04:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.765+0000 7f0a9a83f640 10 monclient: shutdown 2023-11-25T05:04:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.765+0000 7f0a9a83f640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.765+0000 7f0a9a83f640 1 -- 172.21.15.7:0/2105173744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a94142bf0 msgr2=0x7f0a9413ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.765+0000 7f0a9a83f640 1 --2- 172.21.15.7:0/2105173744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a94142bf0 0x7f0a9413ecf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0a940b87d0 tx=0x7f0a8802b9a0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.765+0000 7f0a9a83f640 1 -- 172.21.15.7:0/2105173744 shutdown_connections 2023-11-25T05:04:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.765+0000 7f0a9a83f640 1 --2- 172.21.15.7:0/2105173744 >> v2:172.21.15.7:3300/0 conn(0x7f0a940a8e10 0x7f0a9413f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.765+0000 7f0a9a83f640 1 --2- 172.21.15.7:0/2105173744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a94142bf0 0x7f0a9413ecf0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.765+0000 7f0a9a83f640 1 -- 172.21.15.7:0/2105173744 >> 172.21.15.7:0/2105173744 conn(0x7f0a94061a30 msgr2=0x7f0a940b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.765+0000 7f0a9a83f640 1 -- 172.21.15.7:0/2105173744 shutdown_connections 2023-11-25T05:04:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.765+0000 7f0a9a83f640 1 -- 172.21.15.7:0/2105173744 wait complete. 2023-11-25T05:04:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.765+0000 7f0a9a83f640 10 monclient: build_initial_monmap 2023-11-25T05:04:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.765+0000 7f0a9a83f640 10 monclient: monmap: 2023-11-25T05:04:40.775 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:40.775 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:40.775 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:40.775 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:40.776 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 1 librados: starting msgr at 2023-11-25T05:04:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 1 librados: starting objecter 2023-11-25T05:04:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 1 Processor -- start 2023-11-25T05:04:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 1 -- start start 2023-11-25T05:04:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 1 librados: setting wanted keys 2023-11-25T05:04:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 1 librados: calling monclient init 2023-11-25T05:04:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 10 monclient: init 2023-11-25T05:04:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:04:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a940a8a30 0x7f0a940b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 10 monclient(hunting): picked mon.noname-a con 0x7f0a940a8a30 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0a940a8e10 0x7f0a940b6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 10 monclient(hunting): picked mon.noname-c con 0x7f0a940a8e10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a94142bf0 0x7f0a94145860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 10 monclient(hunting): picked mon.noname-b con 0x7f0a94142bf0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a940b7340 con 0x7f0a940a8a30 2023-11-25T05:04:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a94144100 con 0x7f0a94142bf0 2023-11-25T05:04:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a940b7570 con 0x7f0a940a8e10 2023-11-25T05:04:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.766+0000 7f0a9a83f640 10 monclient(hunting): authenticate will time out at 1407.009788s 2023-11-25T05:04:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.767+0000 7f0a98ddc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a94142bf0 0x7f0a94145860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.767+0000 7f0a937fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0a940a8e10 0x7f0a940b6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.767+0000 7f0a93fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a940a8a30 0x7f0a940b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.767+0000 7f0a98ddc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a94142bf0 0x7f0a94145860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:35348/0 (socket says 172.21.15.7:35348) 2023-11-25T05:04:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.767+0000 7f0a98ddc640 1 -- 172.21.15.7:0/3298192956 learned_addr learned my addr 172.21.15.7:0/3298192956 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.767+0000 7f0a98ddc640 10 monclient(hunting): get_auth_request con 0x7f0a94142bf0 auth_method 0 2023-11-25T05:04:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.767+0000 7f0a98ddc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.767+0000 7f0a98ddc640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.767+0000 7f0a98ddc640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.767+0000 7f0a937fe640 10 monclient(hunting): get_auth_request con 0x7f0a940a8e10 auth_method 0 2023-11-25T05:04:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.767+0000 7f0a937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.767+0000 7f0a937fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.767+0000 7f0a937fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.768+0000 7f0a93fff640 10 monclient(hunting): get_auth_request con 0x7f0a940a8a30 auth_method 0 2023-11-25T05:04:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.768+0000 7f0a93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.768+0000 7f0a93fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.768+0000 7f0a93fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.768+0000 7f0a98ddc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.768+0000 7f0a98ddc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.768+0000 7f0a98ddc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.768+0000 7f0a937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.768+0000 7f0a937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.768+0000 7f0a937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.768+0000 7f0a93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.768+0000 7f0a93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:40.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.768+0000 7f0a93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:40.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.771+0000 7f0a98ddc640 10 monclient(hunting): handle_auth_done global_id 4390 payload 824 2023-11-25T05:04:40.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.771+0000 7f0a98ddc640 1 -- 172.21.15.7:0/3298192956 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0a940a8e10 msgr2=0x7f0a940b6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.771+0000 7f0a98ddc640 1 --2- 172.21.15.7:0/3298192956 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0a940a8e10 0x7f0a940b6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.771+0000 7f0a98ddc640 1 -- 172.21.15.7:0/3298192956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a940a8a30 msgr2=0x7f0a940b61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.771+0000 7f0a98ddc640 1 --2- 172.21.15.7:0/3298192956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a940a8a30 0x7f0a940b61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.771+0000 7f0a98ddc640 10 monclient: _finish_hunting 0 2023-11-25T05:04:40.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.771+0000 7f0a98ddc640 1 monclient: found mon.noname-b 2023-11-25T05:04:40.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.771+0000 7f0a98ddc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:40.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.771+0000 7f0a98ddc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:40.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.771+0000 7f0a98ddc640 1 -- 172.21.15.7:0/3298192956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a880094d0 con 0x7f0a94142bf0 2023-11-25T05:04:40.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.775+0000 7f0a98ddc640 1 --2- 172.21.15.7:0/3298192956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a94142bf0 0x7f0a94145860 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00efc0 tx=0x7f0a8c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:40.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.775+0000 7f0a90ff9640 10 client.?.objecter ms_handle_connect 0x7f0a94142bf0 2023-11-25T05:04:40.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.775+0000 7f0a90ff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:40.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a90ff9640 1 -- 172.21.15.7:0/3298192956 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c014480 con 0x7f0a94142bf0 2023-11-25T05:04:40.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a90ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:40.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a90ff9640 10 monclient: dump: 2023-11-25T05:04:40.783 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.783 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.783 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.783 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.783 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.783 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.783 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.783 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.783 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.783 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a90ff9640 10 monclient: _finish_auth 0 2023-11-25T05:04:40.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a90ff9640 10 monclient: _check_auth_tickets 2023-11-25T05:04:40.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:40.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a90ff9640 1 -- 172.21.15.7:0/3298192956 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8c00f040 con 0x7f0a94142bf0 2023-11-25T05:04:40.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a90ff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:40.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a90ff9640 1 -- 172.21.15.7:0/3298192956 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c014820 con 0x7f0a94142bf0 2023-11-25T05:04:40.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a9a83f640 5 monclient: authenticate success, global_id 4390 2023-11-25T05:04:40.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a9a83f640 10 monclient: _renew_subs 2023-11-25T05:04:40.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a9a83f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:40.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a9a83f640 1 -- 172.21.15.7:0/3298192956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a940b1430 con 0x7f0a94142bf0 2023-11-25T05:04:40.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a927fc640 4 set_mon_vals no callback set 2023-11-25T05:04:40.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a90ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:40.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a90ff9640 10 monclient: dump: 2023-11-25T05:04:40.785 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.785 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.786 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a9a83f640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a9a83f640 10 monclient: _renew_subs 2023-11-25T05:04:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a9a83f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a9a83f640 1 -- 172.21.15.7:0/3298192956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a94145f80 con 0x7f0a94142bf0 2023-11-25T05:04:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.779+0000 7f0a9a83f640 1 librados: init done 2023-11-25T05:04:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.780+0000 7f0a9a83f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.780+0000 7f0a9a83f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.780+0000 7f0a9a83f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.780+0000 7f0a9a83f640 1 -- 172.21.15.7:0/3298192956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a94067ac0 con 0x7f0a94142bf0 2023-11-25T05:04:40.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.797+0000 7f0a90ff9640 1 -- 172.21.15.7:0/3298192956 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0a8c014980 con 0x7f0a94142bf0 2023-11-25T05:04:40.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.798+0000 7f0a90ff9640 1 --2- 172.21.15.7:0/3298192956 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f0a8006c820 0x7f0a8006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.798+0000 7f0a93fff640 1 --2- 172.21.15.7:0/3298192956 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f0a8006c820 0x7f0a8006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.798+0000 7f0a93fff640 10 monclient: get_auth_request con 0x7f0a8006c820 auth_method 0 2023-11-25T05:04:40.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.798+0000 7f0a93fff640 1 --2- 172.21.15.7:0/3298192956 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f0a8006c820 0x7f0a8006ece0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0a84004f60 tx=0x7f0a84007e30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:40.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.798+0000 7f0a90ff9640 10 client.4390.objecter ms_handle_connect 0x7f0a8006c820 2023-11-25T05:04:40.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.800+0000 7f0a90ff9640 1 -- 172.21.15.7:0/3298192956 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0a8c059660 con 0x7f0a94142bf0 2023-11-25T05:04:40.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.800+0000 7f0a90ff9640 10 client.4390.objecter ms_dispatch 0x7f0a94000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:40.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.800+0000 7f0a90ff9640 3 client.4390.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:40.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.800+0000 7f0a90ff9640 3 client.4390.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:40.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.800+0000 7f0a90ff9640 20 client.4390.objecter dump_active .. 0 homeless 2023-11-25T05:04:40.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.856+0000 7f8435d33640 10 monclient: get_monmap_and_config 2023-11-25T05:04:40.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.856+0000 7f8435d33640 10 monclient: build_initial_monmap 2023-11-25T05:04:40.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.856+0000 7f8435d33640 10 monclient: monmap: 2023-11-25T05:04:40.858 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:40.858 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.858 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:40.857252+0000 2023-11-25T05:04:40.858 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:40.857252+0000 2023-11-25T05:04:40.858 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:40.858 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.858 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:40.858 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:40.858 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:40.858 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:40.858 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:40.858 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 1 Processor -- start 2023-11-25T05:04:40.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 1 -- start start 2023-11-25T05:04:40.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 10 monclient: init 2023-11-25T05:04:40.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:40.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-11-25T05:04:40.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84300a8a30 0x7f84301424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 10 monclient(hunting): picked mon.noname-b con 0x7f84300a8a30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:40.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f84300a8e10 0x7f8430142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 10 monclient(hunting): picked mon.noname-a con 0x7f84300a8e10 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84301454a0 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84300aac10 con 0x7f84301454a0 2023-11-25T05:04:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84300aaa10 con 0x7f84300a8e10 2023-11-25T05:04:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84300aaef0 con 0x7f84300a8a30 2023-11-25T05:04:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.857+0000 7f8435d33640 10 monclient(hunting): authenticate will time out at 1407.100702s 2023-11-25T05:04:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.858+0000 7f842effd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f84300a8e10 0x7f8430142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.858+0000 7f842effd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f84300a8e10 0x7f8430142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58558/0 (socket says 172.21.15.7:58558) 2023-11-25T05:04:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.858+0000 7f842effd640 1 -- 172.21.15.7:0/1143588286 learned_addr learned my addr 172.21.15.7:0/1143588286 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.858+0000 7f842effd640 10 monclient(hunting): get_auth_request con 0x7f84300a8e10 auth_method 0 2023-11-25T05:04:40.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.858+0000 7f842effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:40.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.858+0000 7f842effd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:40.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.858+0000 7f842effd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:40.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.859+0000 7f842effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:40.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.859+0000 7f842effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:40.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.859+0000 7f842effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:40.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.861+0000 7f842f7fe640 1 --2- 172.21.15.7:0/1143588286 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84300a8a30 0x7f84301424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.861+0000 7f842f7fe640 10 monclient(hunting): get_auth_request con 0x7f84300a8a30 auth_method 0 2023-11-25T05:04:40.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.861+0000 7f842f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:40.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.861+0000 7f842f7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:40.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.861+0000 7f842f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:40.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.861+0000 7f842effd640 10 monclient(hunting): handle_auth_done global_id 4365 payload 274 2023-11-25T05:04:40.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.861+0000 7f842effd640 1 -- 172.21.15.7:0/1143588286 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84300a8a30 msgr2=0x7f84301424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:40.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.861+0000 7f842effd640 1 --2- 172.21.15.7:0/1143588286 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84300a8a30 0x7f84301424e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.861+0000 7f842effd640 1 -- 172.21.15.7:0/1143588286 >> v1:172.21.15.90:6789/0 conn(0x7f84301454a0 legacy=0x7f8430145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:40.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.861+0000 7f842effd640 10 monclient: _finish_hunting 0 2023-11-25T05:04:40.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.861+0000 7f842effd640 1 monclient: found mon.noname-a 2023-11-25T05:04:40.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.861+0000 7f842effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:40.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.861+0000 7f842effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:40.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.861+0000 7f842effd640 1 -- 172.21.15.7:0/1143588286 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84300a97f0 con 0x7f84300a8e10 2023-11-25T05:04:40.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.863+0000 7f842effd640 1 --2- 172.21.15.7:0/1143588286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84300a8e10 0x7f8430142a20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f84240098a0 tx=0x7f842402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89154a7985460bc7 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:40.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f842e7fc640 1 -- 172.21.15.7:0/1143588286 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424031640 con 0x7f84300a8e10 2023-11-25T05:04:40.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f842e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f842e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:40.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f842e7fc640 10 monclient: dump: 2023-11-25T05:04:40.870 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.871 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.871 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.871 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.871 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.871 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.871 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.871 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.871 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.871 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f842e7fc640 10 monclient: _finish_auth 0 2023-11-25T05:04:40.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f842e7fc640 10 monclient: _check_auth_tickets 2023-11-25T05:04:40.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f842e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:40.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f842e7fc640 1 -- 172.21.15.7:0/1143588286 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84240317a0 con 0x7f84300a8e10 2023-11-25T05:04:40.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f8435d33640 5 monclient: authenticate success, global_id 4365 2023-11-25T05:04:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f842e7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f8435d33640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f8435d33640 4 set_mon_vals no callback set 2023-11-25T05:04:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f842e7fc640 1 -- 172.21.15.7:0/1143588286 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424031b30 con 0x7f84300a8e10 2023-11-25T05:04:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f842e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f842e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f842e7fc640 10 monclient: dump: 2023-11-25T05:04:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.873 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.873 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f8435d33640 10 monclient: shutdown 2023-11-25T05:04:40.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f8435d33640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:40.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f8435d33640 1 -- 172.21.15.7:0/1143588286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84300a8e10 msgr2=0x7f8430142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f8435d33640 1 --2- 172.21.15.7:0/1143588286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84300a8e10 0x7f8430142a20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f84240098a0 tx=0x7f842402b8f0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f8435d33640 1 -- 172.21.15.7:0/1143588286 shutdown_connections 2023-11-25T05:04:40.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f8435d33640 1 --2- 172.21.15.7:0/1143588286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84300a8e10 0x7f8430142a20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f8435d33640 1 --2- 172.21.15.7:0/1143588286 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84300a8a30 0x7f84301424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f8435d33640 1 -- 172.21.15.7:0/1143588286 >> 172.21.15.7:0/1143588286 conn(0x7f8430061a30 msgr2=0x7f84300b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:40.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f8435d33640 1 -- 172.21.15.7:0/1143588286 shutdown_connections 2023-11-25T05:04:40.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.869+0000 7f8435d33640 1 -- 172.21.15.7:0/1143588286 wait complete. 2023-11-25T05:04:40.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.870+0000 7f8435d33640 10 monclient: build_initial_monmap 2023-11-25T05:04:40.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.870+0000 7f8435d33640 10 monclient: monmap: 2023-11-25T05:04:40.876 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:40.876 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:40.876 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:40.876 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:40.876 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:40.876 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.876 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:40.877 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.870+0000 7f8435d33640 1 librados: starting msgr at 2023-11-25T05:04:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.870+0000 7f8435d33640 1 librados: starting objecter 2023-11-25T05:04:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 1 Processor -- start 2023-11-25T05:04:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 1 -- start start 2023-11-25T05:04:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 1 librados: setting wanted keys 2023-11-25T05:04:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 1 librados: calling monclient init 2023-11-25T05:04:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 10 monclient: init 2023-11-25T05:04:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-25T05:04:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84300a8a30 0x7f84301d7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 10 monclient(hunting): picked mon.noname-c con 0x7f84300a8a30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84300a8e10 0x7f843013b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 10 monclient(hunting): picked mon.noname-a con 0x7f84300a8e10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84301454a0 0x7f843013bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 10 monclient(hunting): picked mon.noname-b con 0x7f84301454a0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84300ae840 con 0x7f84300a8e10 2023-11-25T05:04:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84300aeab0 con 0x7f84301454a0 2023-11-25T05:04:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f843013c0f0 con 0x7f84300a8a30 2023-11-25T05:04:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.871+0000 7f8435d33640 10 monclient(hunting): authenticate will time out at 1407.114693s 2023-11-25T05:04:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.872+0000 7f842f7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84300a8a30 0x7f84301d7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.872+0000 7f842effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84300a8e10 0x7f843013b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.872+0000 7f842f7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84300a8a30 0x7f84301d7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41986/0 (socket says 172.21.15.7:41986) 2023-11-25T05:04:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.872+0000 7f842f7fe640 1 -- 172.21.15.7:0/3748309413 learned_addr learned my addr 172.21.15.7:0/3748309413 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.872+0000 7f842f7fe640 10 monclient(hunting): get_auth_request con 0x7f84300a8a30 auth_method 0 2023-11-25T05:04:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.872+0000 7f842f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.872+0000 7f842f7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.872+0000 7f842f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.872+0000 7f842effd640 10 monclient(hunting): get_auth_request con 0x7f84300a8e10 auth_method 0 2023-11-25T05:04:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.872+0000 7f842effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.872+0000 7f842effd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.872+0000 7f842effd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.873+0000 7f842f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.873+0000 7f842f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.873+0000 7f842f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:40.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.873+0000 7f842effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:40.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.873+0000 7f842effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:40.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.873+0000 7f842effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:40.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.876+0000 7f842f7fe640 10 monclient(hunting): handle_auth_done global_id 4331 payload 824 2023-11-25T05:04:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.876+0000 7f842f7fe640 1 -- 172.21.15.7:0/3748309413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84301454a0 msgr2=0x7f843013bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.876+0000 7f842f7fe640 1 --2- 172.21.15.7:0/3748309413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84301454a0 0x7f843013bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.876+0000 7f842f7fe640 1 -- 172.21.15.7:0/3748309413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84300a8e10 msgr2=0x7f843013b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.876+0000 7f842f7fe640 1 --2- 172.21.15.7:0/3748309413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84300a8e10 0x7f843013b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.876+0000 7f842f7fe640 10 monclient: _finish_hunting 0 2023-11-25T05:04:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.876+0000 7f842f7fe640 1 monclient: found mon.noname-c 2023-11-25T05:04:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.876+0000 7f842f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.876+0000 7f842f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:04:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.876+0000 7f842f7fe640 1 -- 172.21.15.7:0/3748309413 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84240094d0 con 0x7f84300a8a30 2023-11-25T05:04:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.878+0000 7f842f7fe640 1 --2- 172.21.15.7:0/3748309413 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84300a8a30 0x7f84301d7060 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f842000d800 tx=0x7f842000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.878+0000 7f8417fff640 10 client.?.objecter ms_handle_connect 0x7f84300a8a30 2023-11-25T05:04:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.878+0000 7f8417fff640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:40.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8417fff640 1 -- 172.21.15.7:0/3748309413 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842000b6c0 con 0x7f84300a8a30 2023-11-25T05:04:40.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8417fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8417fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:40.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8417fff640 10 monclient: dump: 2023-11-25T05:04:40.884 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.884 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.885 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8417fff640 10 monclient: _finish_auth 0 2023-11-25T05:04:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8417fff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8417fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8435d33640 5 monclient: authenticate success, global_id 4331 2023-11-25T05:04:40.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8417fff640 1 -- 172.21.15.7:0/3748309413 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842000b820 con 0x7f84300a8a30 2023-11-25T05:04:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8435d33640 10 monclient: _renew_subs 2023-11-25T05:04:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8435d33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8435d33640 1 -- 172.21.15.7:0/3748309413 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f843013c430 con 0x7f84300a8a30 2023-11-25T05:04:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8417fff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8417fff640 1 -- 172.21.15.7:0/3748309413 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842000bbb0 con 0x7f84300a8a30 2023-11-25T05:04:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8417fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8417fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8417fff640 10 monclient: dump: 2023-11-25T05:04:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:40.887 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f842dffb640 4 set_mon_vals no callback set 2023-11-25T05:04:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8435d33640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8435d33640 10 monclient: _renew_subs 2023-11-25T05:04:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8435d33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8435d33640 1 -- 172.21.15.7:0/3748309413 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84301d7cb0 con 0x7f84300a8a30 2023-11-25T05:04:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.882+0000 7f8435d33640 1 librados: init done 2023-11-25T05:04:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.883+0000 7f8435d33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.883+0000 7f8435d33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.883+0000 7f8435d33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.883+0000 7f8435d33640 1 -- 172.21.15.7:0/3748309413 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8430067ac0 con 0x7f84300a8a30 2023-11-25T05:04:40.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-25T05:04:40.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.904+0000 7f8417fff640 1 -- 172.21.15.7:0/3748309413 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f842000bd10 con 0x7f84300a8a30 2023-11-25T05:04:40.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.904+0000 7f8417fff640 1 --2- 172.21.15.7:0/3748309413 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f841c06c820 0x7f841c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:40.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.904+0000 7f842effd640 1 --2- 172.21.15.7:0/3748309413 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f841c06c820 0x7f841c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:40.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.904+0000 7f842effd640 10 monclient: get_auth_request con 0x7f841c06c820 auth_method 0 2023-11-25T05:04:40.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.905+0000 7f842effd640 1 --2- 172.21.15.7:0/3748309413 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f841c06c820 0x7f841c06ece0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8424036000 tx=0x7f842403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:40.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.905+0000 7f8417fff640 10 client.4331.objecter ms_handle_connect 0x7f841c06c820 2023-11-25T05:04:40.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.907+0000 7f8417fff640 1 -- 172.21.15.7:0/3748309413 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8420059a30 con 0x7f84300a8a30 2023-11-25T05:04:40.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.907+0000 7f8417fff640 10 client.4331.objecter ms_dispatch 0x7f8430000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:40.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.907+0000 7f8417fff640 3 client.4331.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:40.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.907+0000 7f8417fff640 3 client.4331.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:40.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.907+0000 7f8417fff640 20 client.4331.objecter dump_active .. 0 homeless 2023-11-25T05:04:41.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.999+0000 7f2945d83640 10 monclient: get_monmap_and_config 2023-11-25T05:04:41.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.999+0000 7f2945d83640 10 monclient: build_initial_monmap 2023-11-25T05:04:41.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:40.999+0000 7f2945d83640 10 monclient: monmap: 2023-11-25T05:04:41.001 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:41.001 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:41.001 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:41.000476+0000 2023-11-25T05:04:41.001 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:41.000476+0000 2023-11-25T05:04:41.001 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:41.001 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:41.001 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:41.002 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:41.002 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:41.002 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:41.002 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:41.002 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:41.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.000+0000 7f2945d83640 1 Processor -- start 2023-11-25T05:04:41.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.000+0000 7f2945d83640 1 -- start start 2023-11-25T05:04:41.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.000+0000 7f2945d83640 10 monclient: init 2023-11-25T05:04:41.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.000+0000 7f2945d83640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:41.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.000+0000 7f2945d83640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2023-11-25T05:04:41.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.000+0000 7f2945d83640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29400a8910 0x7f294013ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:41.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.000+0000 7f2945d83640 10 monclient(hunting): picked mon.noname-a con 0x7f29400a8910 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:41.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.000+0000 7f2945d83640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f29400a9b70 0x7f294013f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:41.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.000+0000 7f2945d83640 10 monclient(hunting): picked mon.noname-c con 0x7f29400a9b70 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:41.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.001+0000 7f2945d83640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2940141f90 addr v1:172.21.15.90:6789/0 2023-11-25T05:04:41.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.001+0000 7f2945d83640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29400a8cf0 con 0x7f2940141f90 2023-11-25T05:04:41.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.001+0000 7f2945d83640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:41.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.001+0000 7f2945d83640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f294013f6e0 con 0x7f29400a8910 2023-11-25T05:04:41.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.001+0000 7f2945d83640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:41.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.001+0000 7f2945d83640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29400aabd0 con 0x7f29400a9b70 2023-11-25T05:04:41.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.001+0000 7f2945d83640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:41.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.001+0000 7f2945d83640 10 monclient(hunting): authenticate will time out at 1407.243882s 2023-11-25T05:04:41.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.001+0000 7f293f7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29400a8910 0x7f294013ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:41.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.002+0000 7f293f7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29400a8910 0x7f294013ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58568/0 (socket says 172.21.15.7:58568) 2023-11-25T05:04:41.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.002+0000 7f293f7fe640 1 -- 172.21.15.7:0/2015349927 learned_addr learned my addr 172.21.15.7:0/2015349927 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:41.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.002+0000 7f293f7fe640 10 monclient(hunting): get_auth_request con 0x7f29400a8910 auth_method 0 2023-11-25T05:04:41.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.002+0000 7f293f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:41.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.002+0000 7f293f7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:41.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.002+0000 7f293f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:41.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.002+0000 7f293f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:41.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.002+0000 7f293f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:41.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.002+0000 7f293f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:41.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.004+0000 7f293f7fe640 10 monclient(hunting): handle_auth_done global_id 4371 payload 274 2023-11-25T05:04:41.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.004+0000 7f293f7fe640 1 -- 172.21.15.7:0/2015349927 >> v2:172.21.15.90:3300/0 conn(0x7f29400a9b70 msgr2=0x7f294013f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:41.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.004+0000 7f293f7fe640 1 --2- 172.21.15.7:0/2015349927 >> v2:172.21.15.90:3300/0 conn(0x7f29400a9b70 0x7f294013f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.004+0000 7f293f7fe640 1 -- 172.21.15.7:0/2015349927 >> v1:172.21.15.90:6789/0 conn(0x7f2940141f90 legacy=0x7f2940142440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:41.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.004+0000 7f293f7fe640 10 monclient: _finish_hunting 0 2023-11-25T05:04:41.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.004+0000 7f293f7fe640 1 monclient: found mon.noname-a 2023-11-25T05:04:41.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.004+0000 7f293f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:41.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.004+0000 7f293f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:41.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.004+0000 7f293f7fe640 1 -- 172.21.15.7:0/2015349927 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29400aaeb0 con 0x7f29400a8910 2023-11-25T05:04:41.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.006+0000 7f293f7fe640 1 --2- 172.21.15.7:0/2015349927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400a8910 0x7f294013ec60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2940074590 tx=0x7f293002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c3cb826fbc9f15c server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:41.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f293e7fc640 1 -- 172.21.15.7:0/2015349927 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2930031870 con 0x7f29400a8910 2023-11-25T05:04:41.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f293e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:41.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f293e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:41.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f293e7fc640 10 monclient: dump: 2023-11-25T05:04:41.012 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:41.012 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:41.012 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.012 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.012 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:41.012 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:41.013 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:41.013 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:41.013 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:41.013 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:41.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f293e7fc640 10 monclient: _finish_auth 0 2023-11-25T05:04:41.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f293e7fc640 10 monclient: _check_auth_tickets 2023-11-25T05:04:41.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f293e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:41.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f293e7fc640 1 -- 172.21.15.7:0/2015349927 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29300319d0 con 0x7f29400a8910 2023-11-25T05:04:41.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f293e7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:41.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f293e7fc640 1 -- 172.21.15.7:0/2015349927 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2930031d60 con 0x7f29400a8910 2023-11-25T05:04:41.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f2945d83640 5 monclient: authenticate success, global_id 4371 2023-11-25T05:04:41.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f2945d83640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f2945d83640 4 set_mon_vals no callback set 2023-11-25T05:04:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f293e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f293e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.010+0000 7f293e7fc640 10 monclient: dump: 2023-11-25T05:04:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:41.015 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.011+0000 7f2945d83640 10 monclient: shutdown 2023-11-25T05:04:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.011+0000 7f2945d83640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.011+0000 7f2945d83640 1 -- 172.21.15.7:0/2015349927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400a8910 msgr2=0x7f294013ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.011+0000 7f2945d83640 1 --2- 172.21.15.7:0/2015349927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400a8910 0x7f294013ec60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2940074590 tx=0x7f293002b910 comp rx=0 tx=0).stop 2023-11-25T05:04:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.011+0000 7f2945d83640 1 -- 172.21.15.7:0/2015349927 shutdown_connections 2023-11-25T05:04:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.011+0000 7f2945d83640 1 --2- 172.21.15.7:0/2015349927 >> v2:172.21.15.90:3300/0 conn(0x7f29400a9b70 0x7f294013f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.011+0000 7f2945d83640 1 --2- 172.21.15.7:0/2015349927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400a8910 0x7f294013ec60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.011+0000 7f2945d83640 1 -- 172.21.15.7:0/2015349927 >> 172.21.15.7:0/2015349927 conn(0x7f29400b7780 msgr2=0x7f29400b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.011+0000 7f2945d83640 1 -- 172.21.15.7:0/2015349927 shutdown_connections 2023-11-25T05:04:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.011+0000 7f2945d83640 1 -- 172.21.15.7:0/2015349927 wait complete. 2023-11-25T05:04:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.012+0000 7f2945d83640 10 monclient: build_initial_monmap 2023-11-25T05:04:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.012+0000 7f2945d83640 10 monclient: monmap: 2023-11-25T05:04:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:41.019 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.013+0000 7f2945d83640 1 librados: starting msgr at 2023-11-25T05:04:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.013+0000 7f2945d83640 1 librados: starting objecter 2023-11-25T05:04:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.013+0000 7f2945d83640 1 Processor -- start 2023-11-25T05:04:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.013+0000 7f2945d83640 1 -- start start 2023-11-25T05:04:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.013+0000 7f2945d83640 1 librados: setting wanted keys 2023-11-25T05:04:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.013+0000 7f2945d83640 1 librados: calling monclient init 2023-11-25T05:04:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.013+0000 7f2945d83640 10 monclient: init 2023-11-25T05:04:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-25T05:04:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29400a8910 0x7f2940137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:41.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 10 monclient(hunting): picked mon.noname-b con 0x7f29400a8910 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:41.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2940141f90 0x7f29401d6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:41.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 10 monclient(hunting): picked mon.noname-a con 0x7f2940141f90 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:41.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f294013bd80 0x7f29401d7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:41.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 10 monclient(hunting): picked mon.noname-c con 0x7f294013bd80 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:41.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:41.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294013b740 con 0x7f2940141f90 2023-11-25T05:04:41.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:41.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294013bb60 con 0x7f29400a8910 2023-11-25T05:04:41.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:41.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29401d7880 con 0x7f294013bd80 2023-11-25T05:04:41.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:41.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.014+0000 7f2945d83640 10 monclient(hunting): authenticate will time out at 1407.257266s 2023-11-25T05:04:41.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.015+0000 7f293effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2940141f90 0x7f29401d6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:41.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.015+0000 7f293effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2940141f90 0x7f29401d6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58578/0 (socket says 172.21.15.7:58578) 2023-11-25T05:04:41.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.015+0000 7f293effd640 1 -- 172.21.15.7:0/3467999589 learned_addr learned my addr 172.21.15.7:0/3467999589 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:41.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.015+0000 7f293effd640 10 monclient(hunting): get_auth_request con 0x7f2940141f90 auth_method 0 2023-11-25T05:04:41.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.015+0000 7f293effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:41.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.015+0000 7f293effd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:41.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.015+0000 7f293effd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:41.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.016+0000 7f293effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:41.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.016+0000 7f293effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:41.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.016+0000 7f293effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:41.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.019+0000 7f293effd640 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2023-11-25T05:04:41.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.019+0000 7f293effd640 1 -- 172.21.15.7:0/3467999589 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f294013bd80 msgr2=0x7f29401d7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:41.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.019+0000 7f293effd640 1 --2- 172.21.15.7:0/3467999589 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f294013bd80 0x7f29401d7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.019+0000 7f293effd640 1 -- 172.21.15.7:0/3467999589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29400a8910 msgr2=0x7f2940137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:41.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.019+0000 7f293effd640 1 --2- 172.21.15.7:0/3467999589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29400a8910 0x7f2940137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.019+0000 7f293effd640 10 monclient: _finish_hunting 0 2023-11-25T05:04:41.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.019+0000 7f293effd640 1 monclient: found mon.noname-a 2023-11-25T05:04:41.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.019+0000 7f293effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:41.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.019+0000 7f293effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:04:41.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.019+0000 7f293effd640 1 -- 172.21.15.7:0/3467999589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29300094d0 con 0x7f2940141f90 2023-11-25T05:04:41.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.021+0000 7f293effd640 1 --2- 172.21.15.7:0/3467999589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2940141f90 0x7f29401d6e00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f292800b4a0 tx=0x7f292800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:41.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.021+0000 7f2937fff640 10 client.?.objecter ms_handle_connect 0x7f2940141f90 2023-11-25T05:04:41.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.021+0000 7f2937fff640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:41.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2937fff640 1 -- 172.21.15.7:0/3467999589 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2928010610 con 0x7f2940141f90 2023-11-25T05:04:41.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2937fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:41.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2937fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:04:41.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2937fff640 10 monclient: dump: 2023-11-25T05:04:41.027 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:41.027 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:41.027 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.027 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:41.028 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2937fff640 10 monclient: _finish_auth 0 2023-11-25T05:04:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2937fff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2937fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2945d83640 5 monclient: authenticate success, global_id 4374 2023-11-25T05:04:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2937fff640 1 -- 172.21.15.7:0/3467999589 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2928010770 con 0x7f2940141f90 2023-11-25T05:04:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2937fff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2937fff640 1 -- 172.21.15.7:0/3467999589 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2928010b20 con 0x7f2940141f90 2023-11-25T05:04:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2937fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2937fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:04:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2937fff640 10 monclient: dump: 2023-11-25T05:04:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:41.030 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:41.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f293dffb640 4 set_mon_vals no callback set 2023-11-25T05:04:41.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2945d83640 10 monclient: _renew_subs 2023-11-25T05:04:41.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2945d83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:41.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2945d83640 1 -- 172.21.15.7:0/3467999589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294013b360 con 0x7f2940141f90 2023-11-25T05:04:41.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2945d83640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:41.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2945d83640 10 monclient: _renew_subs 2023-11-25T05:04:41.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2945d83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:41.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2945d83640 1 -- 172.21.15.7:0/3467999589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29400ab390 con 0x7f2940141f90 2023-11-25T05:04:41.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.025+0000 7f2945d83640 1 librados: init done 2023-11-25T05:04:41.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.026+0000 7f2945d83640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:41.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.026+0000 7f2945d83640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:41.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.026+0000 7f2945d83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:41.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.026+0000 7f2945d83640 1 -- 172.21.15.7:0/3467999589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29400ac680 con 0x7f2940141f90 2023-11-25T05:04:41.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.031+0000 7f0a90ff9640 1 -- 172.21.15.7:0/3298192956 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a8c0559a0 con 0x7f0a94142bf0 2023-11-25T05:04:41.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.031+0000 7f0a90ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:41.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.031+0000 7f0a90ff9640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.166+0000 7f8417fff640 1 -- 172.21.15.7:0/3748309413 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8420055670 con 0x7f84300a8a30 2023-11-25T05:04:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.166+0000 7f8417fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.166+0000 7f8417fff640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:41.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.209+0000 7f0a9a83f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-11-25T05:04:41.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.210+0000 7f0a9a83f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-11-25T05:04:41.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.210+0000 7f0a9a83f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:41.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.210+0000 7f0a9a83f640 1 -- 172.21.15.7:0/3298192956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f0a941d6e60 con 0x7f0a94142bf0 2023-11-25T05:04:41.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.213+0000 7f0a90ff9640 1 -- 172.21.15.7:0/3298192956 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0a941d6e60 con 0x7f0a94142bf0 2023-11-25T05:04:41.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.213+0000 7f0a90ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-11-25T05:04:41.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.213+0000 7f0a90ff9640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:41.214 INFO:teuthology.orchestra.run.smithi007.stdout:64424509446 2023-11-25T05:04:41.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 10 librados: watch_flush enter 2023-11-25T05:04:41.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 10 librados: watch_flush exit 2023-11-25T05:04:41.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 20 client.4390.objecter shutdown clearing up homeless session... 2023-11-25T05:04:41.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 10 client.4390.objecter successfully canceled tick 2023-11-25T05:04:41.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 1 -- 172.21.15.7:0/3298192956 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f0a8006c820 msgr2=0x7f0a8006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:41.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 1 --2- 172.21.15.7:0/3298192956 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f0a8006c820 0x7f0a8006ece0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0a84004f60 tx=0x7f0a84007e30 comp rx=0 tx=0).stop 2023-11-25T05:04:41.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 10 monclient: shutdown 2023-11-25T05:04:41.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:41.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 1 -- 172.21.15.7:0/3298192956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a94142bf0 msgr2=0x7f0a94145860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:41.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 1 --2- 172.21.15.7:0/3298192956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a94142bf0 0x7f0a94145860 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00efc0 tx=0x7f0a8c00c490 comp rx=0 tx=0).stop 2023-11-25T05:04:41.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 1 -- 172.21.15.7:0/3298192956 shutdown_connections 2023-11-25T05:04:41.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 1 --2- 172.21.15.7:0/3298192956 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f0a8006c820 0x7f0a8006ece0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 1 --2- 172.21.15.7:0/3298192956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a94142bf0 0x7f0a94145860 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 1 --2- 172.21.15.7:0/3298192956 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0a940a8e10 0x7f0a940b6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 1 --2- 172.21.15.7:0/3298192956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a940a8a30 0x7f0a940b61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 1 -- 172.21.15.7:0/3298192956 >> 172.21.15.7:0/3298192956 conn(0x7f0a94061a30 msgr2=0x7f0a9413c040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:41.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 1 -- 172.21.15.7:0/3298192956 shutdown_connections 2023-11-25T05:04:41.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 1 -- 172.21.15.7:0/3298192956 wait complete. 2023-11-25T05:04:41.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.215+0000 7f0a9a83f640 1 librados: shutdown 2023-11-25T05:04:41.227 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.6 2023-11-25T05:04:41.228 DEBUG:teuthology.parallel:result is None 2023-11-25T05:04:41.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.307+0000 7f2937fff640 1 -- 172.21.15.7:0/3467999589 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2928010c80 con 0x7f2940141f90 2023-11-25T05:04:41.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.307+0000 7f2937fff640 1 --2- 172.21.15.7:0/3467999589 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f292006c820 0x7f292006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:41.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.308+0000 7f293f7fe640 1 --2- 172.21.15.7:0/3467999589 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f292006c820 0x7f292006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:41.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.308+0000 7f293f7fe640 10 monclient: get_auth_request con 0x7f292006c820 auth_method 0 2023-11-25T05:04:41.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.308+0000 7f293f7fe640 1 --2- 172.21.15.7:0/3467999589 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f292006c820 0x7f292006ece0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2930036000 tx=0x7f2930030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:41.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.308+0000 7f2937fff640 10 client.4374.objecter ms_handle_connect 0x7f292006c820 2023-11-25T05:04:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.322+0000 7f2937fff640 1 -- 172.21.15.7:0/3467999589 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2928058970 con 0x7f2940141f90 2023-11-25T05:04:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.322+0000 7f2937fff640 10 client.4374.objecter ms_dispatch 0x7f2940000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.322+0000 7f2937fff640 3 client.4374.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.322+0000 7f2937fff640 3 client.4374.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.322+0000 7f2937fff640 20 client.4374.objecter dump_active .. 0 homeless 2023-11-25T05:04:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.322+0000 7f2937fff640 1 -- 172.21.15.7:0/3467999589 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29280545b0 con 0x7f2940141f90 2023-11-25T05:04:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.322+0000 7f2937fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.322+0000 7f2937fff640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:41.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.362+0000 7f8435d33640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-11-25T05:04:41.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.362+0000 7f8435d33640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-11-25T05:04:41.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.362+0000 7f8435d33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:04:41.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.362+0000 7f8435d33640 1 -- 172.21.15.7:0/3748309413 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f8430075b60 con 0x7f84300a8a30 2023-11-25T05:04:41.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.368+0000 7f8417fff640 1 -- 172.21.15.7:0/3748309413 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8420058c90 con 0x7f84300a8a30 2023-11-25T05:04:41.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.368+0000 7f8417fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-11-25T05:04:41.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.368+0000 7f8417fff640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:41.370 INFO:teuthology.orchestra.run.smithi007.stdout:68719476741 2023-11-25T05:04:41.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 10 librados: watch_flush enter 2023-11-25T05:04:41.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 10 librados: watch_flush exit 2023-11-25T05:04:41.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 20 client.4331.objecter shutdown clearing up homeless session... 2023-11-25T05:04:41.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 10 client.4331.objecter successfully canceled tick 2023-11-25T05:04:41.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 1 -- 172.21.15.7:0/3748309413 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f841c06c820 msgr2=0x7f841c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:41.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 1 --2- 172.21.15.7:0/3748309413 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f841c06c820 0x7f841c06ece0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8424036000 tx=0x7f842403c040 comp rx=0 tx=0).stop 2023-11-25T05:04:41.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 10 monclient: shutdown 2023-11-25T05:04:41.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:41.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 1 -- 172.21.15.7:0/3748309413 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84300a8a30 msgr2=0x7f84301d7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:41.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 1 --2- 172.21.15.7:0/3748309413 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84300a8a30 0x7f84301d7060 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f842000d800 tx=0x7f842000dcc0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 1 -- 172.21.15.7:0/3748309413 shutdown_connections 2023-11-25T05:04:41.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 1 --2- 172.21.15.7:0/3748309413 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f841c06c820 0x7f841c06ece0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 1 --2- 172.21.15.7:0/3748309413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84301454a0 0x7f843013bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 1 --2- 172.21.15.7:0/3748309413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84300a8e10 0x7f843013b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 1 --2- 172.21.15.7:0/3748309413 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84300a8a30 0x7f84301d7060 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 1 -- 172.21.15.7:0/3748309413 >> 172.21.15.7:0/3748309413 conn(0x7f8430061a30 msgr2=0x7f843013cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:41.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 1 -- 172.21.15.7:0/3748309413 shutdown_connections 2023-11-25T05:04:41.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 1 -- 172.21.15.7:0/3748309413 wait complete. 2023-11-25T05:04:41.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.370+0000 7f8435d33640 1 librados: shutdown 2023-11-25T05:04:41.386 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476741 for osd.0 2023-11-25T05:04:41.386 DEBUG:teuthology.parallel:result is None 2023-11-25T05:04:41.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.512+0000 7f2945d83640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-11-25T05:04:41.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.512+0000 7f2945d83640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-11-25T05:04:41.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.512+0000 7f2945d83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:04:41.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.512+0000 7f2945d83640 1 -- 172.21.15.7:0/3467999589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f29400ac680 con 0x7f2940141f90 2023-11-25T05:04:41.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.545+0000 7f2937fff640 1 -- 172.21.15.7:0/3467999589 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f29400ac680 con 0x7f2940141f90 2023-11-25T05:04:41.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.545+0000 7f2937fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-11-25T05:04:41.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.545+0000 7f2937fff640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:41.552 INFO:teuthology.orchestra.run.smithi007.stdout:68719476741 2023-11-25T05:04:41.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 10 librados: watch_flush enter 2023-11-25T05:04:41.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 10 librados: watch_flush exit 2023-11-25T05:04:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 20 client.4374.objecter shutdown clearing up homeless session... 2023-11-25T05:04:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 10 client.4374.objecter successfully canceled tick 2023-11-25T05:04:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 1 -- 172.21.15.7:0/3467999589 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f292006c820 msgr2=0x7f292006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 1 --2- 172.21.15.7:0/3467999589 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f292006c820 0x7f292006ece0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2930036000 tx=0x7f2930030000 comp rx=0 tx=0).stop 2023-11-25T05:04:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 10 monclient: shutdown 2023-11-25T05:04:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 1 -- 172.21.15.7:0/3467999589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2940141f90 msgr2=0x7f29401d6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 1 --2- 172.21.15.7:0/3467999589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2940141f90 0x7f29401d6e00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f292800b4a0 tx=0x7f292800b960 comp rx=0 tx=0).stop 2023-11-25T05:04:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 1 -- 172.21.15.7:0/3467999589 shutdown_connections 2023-11-25T05:04:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 1 --2- 172.21.15.7:0/3467999589 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f292006c820 0x7f292006ece0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 1 --2- 172.21.15.7:0/3467999589 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f294013bd80 0x7f29401d7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 1 --2- 172.21.15.7:0/3467999589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2940141f90 0x7f29401d6e00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 1 --2- 172.21.15.7:0/3467999589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29400a8910 0x7f2940137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 1 -- 172.21.15.7:0/3467999589 >> 172.21.15.7:0/3467999589 conn(0x7f29400b7780 msgr2=0x7f29400b7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 1 -- 172.21.15.7:0/3467999589 shutdown_connections 2023-11-25T05:04:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 1 -- 172.21.15.7:0/3467999589 wait complete. 2023-11-25T05:04:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.547+0000 7f2945d83640 1 librados: shutdown 2023-11-25T05:04:41.561 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476741 for osd.2 2023-11-25T05:04:41.562 DEBUG:teuthology.parallel:result is None 2023-11-25T05:04:41.562 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-25T05:04:41.562 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-25T05:04:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.667+0000 7f8678989640 10 monclient: get_monmap_and_config 2023-11-25T05:04:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.667+0000 7f8678989640 10 monclient: build_initial_monmap 2023-11-25T05:04:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.667+0000 7f8678989640 10 monclient: monmap: 2023-11-25T05:04:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:41.668678+0000 2023-11-25T05:04:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:41.668678+0000 2023-11-25T05:04:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:41.670 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:41.670 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:41.670 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:41.670 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:41.670 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:41.670 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:41.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.668+0000 7f8678989640 1 Processor -- start 2023-11-25T05:04:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.668+0000 7f8678989640 1 -- start start 2023-11-25T05:04:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.668+0000 7f8678989640 10 monclient: init 2023-11-25T05:04:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.668+0000 7f8678989640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.668+0000 7f8678989640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2023-11-25T05:04:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8678989640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86740a8910 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8678989640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f86740a9b70 0x7f867413f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8678989640 10 monclient(hunting): picked mon.noname-b con 0x7f86740a9b70 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8678989640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8674141f90 0x7f86741423b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8678989640 10 monclient(hunting): picked mon.noname-c con 0x7f8674141f90 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8678989640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86740a8cf0 con 0x7f86740a8910 2023-11-25T05:04:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8678989640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8678989640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86740aabd0 con 0x7f8674141f90 2023-11-25T05:04:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8678989640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8678989640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86740aaeb0 con 0x7f86740a9b70 2023-11-25T05:04:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8678989640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8678989640 10 monclient(hunting): authenticate will time out at 1407.911919s 2023-11-25T05:04:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8669d9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f86740a9b70 0x7f867413f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8669d9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f86740a9b70 0x7f867413f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:42014/0 (socket says 172.21.15.7:42014) 2023-11-25T05:04:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8669d9b640 1 -- 172.21.15.7:0/208901464 learned_addr learned my addr 172.21.15.7:0/208901464 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8669d9b640 10 monclient(hunting): get_auth_request con 0x7f86740a9b70 auth_method 0 2023-11-25T05:04:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8669d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8669d9b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.669+0000 7f8669d9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.670+0000 7f8669d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.670+0000 7f8669d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.670+0000 7f8669d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.671+0000 7f8672d9d640 1 --2- 172.21.15.7:0/208901464 >> v2:172.21.15.90:3300/0 conn(0x7f8674141f90 0x7f86741423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.671+0000 7f8672d9d640 10 monclient(hunting): get_auth_request con 0x7f8674141f90 auth_method 0 2023-11-25T05:04:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.671+0000 7f8672d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.671+0000 7f8672d9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.671+0000 7f8672d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.672+0000 7f8669d9b640 10 monclient(hunting): handle_auth_done global_id 4334 payload 274 2023-11-25T05:04:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.672+0000 7f8669d9b640 1 -- 172.21.15.7:0/208901464 >> v2:172.21.15.90:3300/0 conn(0x7f8674141f90 msgr2=0x7f86741423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.672+0000 7f8669d9b640 1 --2- 172.21.15.7:0/208901464 >> v2:172.21.15.90:3300/0 conn(0x7f8674141f90 0x7f86741423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.672+0000 7f8669d9b640 1 -- 172.21.15.7:0/208901464 >> v1:172.21.15.7:6789/0 conn(0x7f86740a8910 legacy=0x7f867413ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.672+0000 7f8669d9b640 10 monclient: _finish_hunting 0 2023-11-25T05:04:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.672+0000 7f8669d9b640 1 monclient: found mon.noname-b 2023-11-25T05:04:41.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.672+0000 7f8669d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:41.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.672+0000 7f8669d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:04:41.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.672+0000 7f8669d9b640 1 -- 172.21.15.7:0/208901464 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86740a9720 con 0x7f86740a9b70 2023-11-25T05:04:41.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.672+0000 7f8672d9d640 1 --2- 172.21.15.7:0/208901464 >> v2:172.21.15.90:3300/0 conn(0x7f8674141f90 0x7f86741423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-25T05:04:41.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.674+0000 7f8669d9b640 1 --2- 172.21.15.7:0/208901464 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f86740a9b70 0x7f867413f370 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f865c009870 tx=0x7f865c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34cf394b4e38ebcb server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8671d9b640 1 -- 172.21.15.7:0/208901464 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c031640 con 0x7f86740a9b70 2023-11-25T05:04:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8671d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8671d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8671d9b640 10 monclient: dump: 2023-11-25T05:04:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:41.680 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8671d9b640 10 monclient: _finish_auth 0 2023-11-25T05:04:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8671d9b640 10 monclient: _check_auth_tickets 2023-11-25T05:04:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8671d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8671d9b640 1 -- 172.21.15.7:0/208901464 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865c0317a0 con 0x7f86740a9b70 2023-11-25T05:04:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8678989640 5 monclient: authenticate success, global_id 4334 2023-11-25T05:04:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8678989640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:04:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8671d9b640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8678989640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8671d9b640 1 -- 172.21.15.7:0/208901464 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c031b30 con 0x7f86740a9b70 2023-11-25T05:04:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8678989640 4 set_mon_vals no callback set 2023-11-25T05:04:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8671d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8671d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.677+0000 7f8671d9b640 10 monclient: dump: 2023-11-25T05:04:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:41.682 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.678+0000 7f8678989640 10 monclient: shutdown 2023-11-25T05:04:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.678+0000 7f8678989640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.678+0000 7f8678989640 1 -- 172.21.15.7:0/208901464 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f86740a9b70 msgr2=0x7f867413f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.678+0000 7f8678989640 1 --2- 172.21.15.7:0/208901464 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f86740a9b70 0x7f867413f370 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f865c009870 tx=0x7f865c02b8f0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.678+0000 7f8678989640 1 -- 172.21.15.7:0/208901464 shutdown_connections 2023-11-25T05:04:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.678+0000 7f8678989640 1 --2- 172.21.15.7:0/208901464 >> v2:172.21.15.90:3300/0 conn(0x7f8674141f90 0x7f86741423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.678+0000 7f8678989640 1 --2- 172.21.15.7:0/208901464 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f86740a9b70 0x7f867413f370 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.678+0000 7f8678989640 1 -- 172.21.15.7:0/208901464 >> 172.21.15.7:0/208901464 conn(0x7f86740b7780 msgr2=0x7f86740b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.678+0000 7f8678989640 1 -- 172.21.15.7:0/208901464 shutdown_connections 2023-11-25T05:04:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.678+0000 7f8678989640 1 -- 172.21.15.7:0/208901464 wait complete. 2023-11-25T05:04:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.679+0000 7f8678989640 10 monclient: build_initial_monmap 2023-11-25T05:04:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.679+0000 7f8678989640 10 monclient: monmap: 2023-11-25T05:04:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:41.687 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.679+0000 7f8678989640 1 librados: starting msgr at 2023-11-25T05:04:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.679+0000 7f8678989640 1 librados: starting objecter 2023-11-25T05:04:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 1 Processor -- start 2023-11-25T05:04:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 1 -- start start 2023-11-25T05:04:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 1 librados: setting wanted keys 2023-11-25T05:04:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 1 librados: calling monclient init 2023-11-25T05:04:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 10 monclient: init 2023-11-25T05:04:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:04:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f86740a8910 0x7f8674137e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 10 monclient(hunting): picked mon.noname-c con 0x7f86740a8910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8674141f90 0x7f867413bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 10 monclient(hunting): picked mon.noname-b con 0x7f8674141f90 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8674138ee0 0x7f8674139360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 10 monclient(hunting): picked mon.noname-a con 0x7f8674138ee0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867413b480 con 0x7f8674138ee0 2023-11-25T05:04:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867413ad80 con 0x7f8674141f90 2023-11-25T05:04:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86741398a0 con 0x7f86740a8910 2023-11-25T05:04:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.680+0000 7f8678989640 10 monclient(hunting): authenticate will time out at 1407.923561s 2023-11-25T05:04:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f867259c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f86740a8910 0x7f8674137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f8672d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8674138ee0 0x7f8674139360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f8669d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8674141f90 0x7f867413bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f867259c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f86740a8910 0x7f8674137e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:42026/0 (socket says 172.21.15.7:42026) 2023-11-25T05:04:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f867259c640 1 -- 172.21.15.7:0/407731873 learned_addr learned my addr 172.21.15.7:0/407731873 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f867259c640 10 monclient(hunting): get_auth_request con 0x7f86740a8910 auth_method 0 2023-11-25T05:04:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f867259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f867259c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f867259c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f8669d9b640 10 monclient(hunting): get_auth_request con 0x7f8674141f90 auth_method 0 2023-11-25T05:04:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f8669d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f8669d9b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f8669d9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f8672d9d640 10 monclient(hunting): get_auth_request con 0x7f8674138ee0 auth_method 0 2023-11-25T05:04:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f8672d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f8672d9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.681+0000 7f8672d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.682+0000 7f867259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.682+0000 7f867259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.682+0000 7f867259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.682+0000 7f8669d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.682+0000 7f8669d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.682+0000 7f8669d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.682+0000 7f8672d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.682+0000 7f8672d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:41.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.682+0000 7f8672d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:41.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.684+0000 7f8669d9b640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2023-11-25T05:04:41.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.684+0000 7f8669d9b640 1 -- 172.21.15.7:0/407731873 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f86740a8910 msgr2=0x7f8674137e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:41.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.684+0000 7f8669d9b640 1 --2- 172.21.15.7:0/407731873 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f86740a8910 0x7f8674137e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.684+0000 7f8669d9b640 1 -- 172.21.15.7:0/407731873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8674138ee0 msgr2=0x7f8674139360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:41.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.684+0000 7f8669d9b640 1 --2- 172.21.15.7:0/407731873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8674138ee0 0x7f8674139360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:41.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.685+0000 7f8669d9b640 10 monclient: _finish_hunting 0 2023-11-25T05:04:41.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.685+0000 7f8669d9b640 1 monclient: found mon.noname-b 2023-11-25T05:04:41.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.685+0000 7f8669d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:41.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.685+0000 7f8669d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:41.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.685+0000 7f8669d9b640 1 -- 172.21.15.7:0/407731873 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c0094d0 con 0x7f8674141f90 2023-11-25T05:04:41.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.687+0000 7f8669d9b640 1 --2- 172.21.15.7:0/407731873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8674141f90 0x7f867413bbb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f865c033000 tx=0x7f865c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.687+0000 7f866b7fe640 10 client.?.objecter ms_handle_connect 0x7f8674141f90 2023-11-25T05:04:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.687+0000 7f866b7fe640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f866b7fe640 1 -- 172.21.15.7:0/407731873 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c031ea0 con 0x7f8674141f90 2023-11-25T05:04:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f866b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f866b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f866b7fe640 10 monclient: dump: 2023-11-25T05:04:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.695 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:41.696 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f866b7fe640 10 monclient: _finish_auth 0 2023-11-25T05:04:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f866b7fe640 10 monclient: _check_auth_tickets 2023-11-25T05:04:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f866b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f866b7fe640 1 -- 172.21.15.7:0/407731873 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865c03ac80 con 0x7f8674141f90 2023-11-25T05:04:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f8678989640 5 monclient: authenticate success, global_id 4396 2023-11-25T05:04:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f866b7fe640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:41.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f866b7fe640 1 -- 172.21.15.7:0/407731873 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c0443f0 con 0x7f8674141f90 2023-11-25T05:04:41.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f8678989640 10 monclient: _renew_subs 2023-11-25T05:04:41.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f8678989640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:41.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f8678989640 1 -- 172.21.15.7:0/407731873 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f867413c0f0 con 0x7f8674141f90 2023-11-25T05:04:41.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f866b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:41.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f866b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:41.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f866b7fe640 10 monclient: dump: 2023-11-25T05:04:41.697 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:41.697 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:41.697 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.697 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:41.697 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:41.698 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:41.698 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:41.698 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:41.698 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:41.698 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:41.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f867159a640 4 set_mon_vals no callback set 2023-11-25T05:04:41.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f8678989640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:41.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f8678989640 10 monclient: _renew_subs 2023-11-25T05:04:41.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f8678989640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:41.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f8678989640 1 -- 172.21.15.7:0/407731873 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86741d6da0 con 0x7f8674141f90 2023-11-25T05:04:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.691+0000 7f8678989640 1 librados: init done 2023-11-25T05:04:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.692+0000 7f8678989640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.692+0000 7f8678989640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.692+0000 7f8678989640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.692+0000 7f8678989640 1 -- 172.21.15.7:0/407731873 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f867413b8d0 con 0x7f8674141f90 2023-11-25T05:04:41.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.933+0000 7f866b7fe640 1 -- 172.21.15.7:0/407731873 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f865c02f050 con 0x7f8674141f90 2023-11-25T05:04:41.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.933+0000 7f866b7fe640 1 --2- 172.21.15.7:0/407731873 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f865806c800 0x7f865806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:41.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.933+0000 7f867259c640 1 --2- 172.21.15.7:0/407731873 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f865806c800 0x7f865806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:41.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.933+0000 7f867259c640 10 monclient: get_auth_request con 0x7f865806c800 auth_method 0 2023-11-25T05:04:41.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.934+0000 7f867259c640 1 --2- 172.21.15.7:0/407731873 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f865806c800 0x7f865806ecc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f86640018b0 tx=0x7f8664009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:41.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.934+0000 7f866b7fe640 10 client.4396.objecter ms_handle_connect 0x7f865806c800 2023-11-25T05:04:41.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.936+0000 7f866b7fe640 1 -- 172.21.15.7:0/407731873 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f865c07a3e0 con 0x7f8674141f90 2023-11-25T05:04:41.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.936+0000 7f866b7fe640 10 client.4396.objecter ms_dispatch 0x7f8674000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:41.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.936+0000 7f866b7fe640 3 client.4396.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.936+0000 7f866b7fe640 3 client.4396.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:41.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.936+0000 7f866b7fe640 20 client.4396.objecter dump_active .. 0 homeless 2023-11-25T05:04:41.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.945+0000 7f866b7fe640 1 -- 172.21.15.7:0/407731873 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f865c076720 con 0x7f8674141f90 2023-11-25T05:04:41.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.945+0000 7f866b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:41.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:41.945+0000 7f866b7fe640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:42.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.120+0000 7f8678989640 1 -- 172.21.15.7:0/407731873 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f867413b8d0 con 0x7f865806c800 2023-11-25T05:04:42.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.123+0000 7f866b7fe640 1 -- 172.21.15.7:0/407731873 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48013 (secure 0 0 0) 0x7f867413b8d0 con 0x7f865806c800 2023-11-25T05:04:42.125 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:04:42.125 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-25T05:04:42.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.124+0000 7f8678989640 10 librados: watch_flush enter 2023-11-25T05:04:42.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.124+0000 7f8678989640 10 librados: watch_flush exit 2023-11-25T05:04:42.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.124+0000 7f8678989640 20 client.4396.objecter shutdown clearing up homeless session... 2023-11-25T05:04:42.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.124+0000 7f8678989640 10 client.4396.objecter successfully canceled tick 2023-11-25T05:04:42.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.124+0000 7f8678989640 1 -- 172.21.15.7:0/407731873 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f865806c800 msgr2=0x7f865806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:42.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.124+0000 7f8678989640 1 --2- 172.21.15.7:0/407731873 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f865806c800 0x7f865806ecc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f86640018b0 tx=0x7f8664009040 comp rx=0 tx=0).stop 2023-11-25T05:04:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.124+0000 7f8678989640 10 monclient: shutdown 2023-11-25T05:04:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.124+0000 7f8678989640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.124+0000 7f8678989640 1 -- 172.21.15.7:0/407731873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8674141f90 msgr2=0x7f867413bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.124+0000 7f8678989640 1 --2- 172.21.15.7:0/407731873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8674141f90 0x7f867413bbb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f865c033000 tx=0x7f865c0046c0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.125+0000 7f8678989640 1 -- 172.21.15.7:0/407731873 shutdown_connections 2023-11-25T05:04:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.125+0000 7f8678989640 1 --2- 172.21.15.7:0/407731873 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f865806c800 0x7f865806ecc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.125+0000 7f8678989640 1 --2- 172.21.15.7:0/407731873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8674138ee0 0x7f8674139360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.125+0000 7f8678989640 1 --2- 172.21.15.7:0/407731873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8674141f90 0x7f867413bbb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.125+0000 7f8678989640 1 --2- 172.21.15.7:0/407731873 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f86740a8910 0x7f8674137e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.125+0000 7f8678989640 1 -- 172.21.15.7:0/407731873 >> 172.21.15.7:0/407731873 conn(0x7f86740b7780 msgr2=0x7f867413c860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.125+0000 7f8678989640 1 -- 172.21.15.7:0/407731873 shutdown_connections 2023-11-25T05:04:42.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.125+0000 7f8678989640 1 -- 172.21.15.7:0/407731873 wait complete. 2023-11-25T05:04:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.125+0000 7f8678989640 1 librados: shutdown 2023-11-25T05:04:42.137 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-11-25T05:04:41.373571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218832,"kb_used_data":2468,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755888,"statfs":{"total":773094113280,"available":772870029312,"internally_reserved":0,"allocated":2527232,"data_stored":1395084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":58,"apply_latency_ms":58,"commit_latency_ns":58000000,"apply_latency_ns":58000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001536"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:36.798887+0000","last_change":"2023-11-25T05:04:33.582978+0000","last_active":"2023-11-25T05:04:36.798887+0000","last_peered":"2023-11-25T05:04:36.798887+0000","last_clean":"2023-11-25T05:04:36.798887+0000","last_became_active":"2023-11-25T05:04:33.577510+0000","last_became_peered":"2023-11-25T05:04:33.577510+0000","last_unstale":"2023-11-25T05:04:36.798887+0000","last_undegraded":"2023-11-25T05:04:36.798887+0000","last_fullsized":"2023-11-25T05:04:36.798887+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:53:20.091826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:36.765181+0000","last_change":"2023-11-25T05:04:33.645680+0000","last_active":"2023-11-25T05:04:36.765181+0000","last_peered":"2023-11-25T05:04:36.765181+0000","last_clean":"2023-11-25T05:04:36.765181+0000","last_became_active":"2023-11-25T05:04:33.610776+0000","last_became_peered":"2023-11-25T05:04:33.610776+0000","last_unstale":"2023-11-25T05:04:36.765181+0000","last_undegraded":"2023-11-25T05:04:36.765181+0000","last_fullsized":"2023-11-25T05:04:36.765181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:02:08.013900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:36.789348+0000","last_change":"2023-11-25T05:04:33.350255+0000","last_active":"2023-11-25T05:04:36.789348+0000","last_peered":"2023-11-25T05:04:36.789348+0000","last_clean":"2023-11-25T05:04:36.789348+0000","last_became_active":"2023-11-25T05:04:33.337158+0000","last_became_peered":"2023-11-25T05:04:33.337158+0000","last_unstale":"2023-11-25T05:04:36.789348+0000","last_undegraded":"2023-11-25T05:04:36.789348+0000","last_fullsized":"2023-11-25T05:04:36.789348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:25:34.365788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:35.704622+0000","last_change":"2023-11-25T05:04:33.396056+0000","last_active":"2023-11-25T05:04:35.704622+0000","last_peered":"2023-11-25T05:04:35.704622+0000","last_clean":"2023-11-25T05:04:35.704622+0000","last_became_active":"2023-11-25T05:04:33.388849+0000","last_became_peered":"2023-11-25T05:04:33.388849+0000","last_unstale":"2023-11-25T05:04:35.704622+0000","last_undegraded":"2023-11-25T05:04:35.704622+0000","last_fullsized":"2023-11-25T05:04:35.704622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:11:08.763309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:36.762061+0000","last_change":"2023-11-25T05:04:33.381810+0000","last_active":"2023-11-25T05:04:36.762061+0000","last_peered":"2023-11-25T05:04:36.762061+0000","last_clean":"2023-11-25T05:04:36.762061+0000","last_became_active":"2023-11-25T05:04:33.376375+0000","last_became_peered":"2023-11-25T05:04:33.376375+0000","last_unstale":"2023-11-25T05:04:36.762061+0000","last_undegraded":"2023-11-25T05:04:36.762061+0000","last_fullsized":"2023-11-25T05:04:36.762061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:41:46.038487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-25T05:04:34.697761+0000","last_change":"2023-11-25T05:04:33.238092+0000","last_active":"2023-11-25T05:04:34.697761+0000","last_peered":"2023-11-25T05:04:34.697761+0000","last_clean":"2023-11-25T05:04:34.697761+0000","last_became_active":"2023-11-25T05:04:33.196533+0000","last_became_peered":"2023-11-25T05:04:33.196533+0000","last_unstale":"2023-11-25T05:04:34.697761+0000","last_undegraded":"2023-11-25T05:04:34.697761+0000","last_fullsized":"2023-11-25T05:04:34.697761+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:15:44.127680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:36.775112+0000","last_change":"2023-11-25T05:04:33.294027+0000","last_active":"2023-11-25T05:04:36.775112+0000","last_peered":"2023-11-25T05:04:36.775112+0000","last_clean":"2023-11-25T05:04:36.775112+0000","last_became_active":"2023-11-25T05:04:33.242068+0000","last_became_peered":"2023-11-25T05:04:33.242068+0000","last_unstale":"2023-11-25T05:04:36.775112+0000","last_undegraded":"2023-11-25T05:04:36.775112+0000","last_fullsized":"2023-11-25T05:04:36.775112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:37:54.756624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:36.784564+0000","last_change":"2023-11-25T05:04:33.784328+0000","last_active":"2023-11-25T05:04:36.784564+0000","last_peered":"2023-11-25T05:04:36.784564+0000","last_clean":"2023-11-25T05:04:36.784564+0000","last_became_active":"2023-11-25T05:04:33.778042+0000","last_became_peered":"2023-11-25T05:04:33.778042+0000","last_unstale":"2023-11-25T05:04:36.784564+0000","last_undegraded":"2023-11-25T05:04:36.784564+0000","last_fullsized":"2023-11-25T05:04:36.784564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:40:38.193703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":106,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:37.659883+0000","last_change":"2023-11-25T05:04:28.241367+0000","last_active":"2023-11-25T05:04:37.659883+0000","last_peered":"2023-11-25T05:04:37.659883+0000","last_clean":"2023-11-25T05:04:37.659883+0000","last_became_active":"2023-11-25T05:04:27.388156+0000","last_became_peered":"2023-11-25T05:04:27.388156+0000","last_unstale":"2023-11-25T05:04:37.659883+0000","last_undegraded":"2023-11-25T05:04:37.659883+0000","last_fullsized":"2023-11-25T05:04:37.659883+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:22.310975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:22.310975+0000","last_clean_scrub_stamp":"2023-11-25T05:04:22.310975+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:50:37.122665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":16,"seq":68719476742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8249999999999993}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.350000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.161999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.564}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.702}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.239000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.321999999999999}]}]},{"osd":1,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":618496,"data_stored":473484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.693}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.5069999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.724}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.3490000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.895}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.904}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.871}]}]},{"osd":3,"up_from":16,"seq":68719476742,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.898}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.305}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.527000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4280000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.109000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.214}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.658999999999999}]}]},{"osd":0,"up_from":16,"seq":68719476742,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9130000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.903}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.349}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.143999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.132}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.686}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.765999999999998}]}]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.373000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6449999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.619999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.332000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.343999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.07}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.684999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9510000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4559999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.611000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.698}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.23}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.571000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.534000000000001}]}]},{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":69896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":142.904}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.012}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.472000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":172.67400000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.224}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.960000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.268000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":618496,"data_stored":473484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8810000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.981}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.585000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.914000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.41}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.097999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.419}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-25T05:04:42.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-25T05:04:42.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.242+0000 7fc1b1b05640 10 monclient: get_monmap_and_config 2023-11-25T05:04:42.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.242+0000 7fc1b1b05640 10 monclient: build_initial_monmap 2023-11-25T05:04:42.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.243+0000 7fc1b1b05640 10 monclient: monmap: 2023-11-25T05:04:42.244 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:42.244 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:42.245 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:42.243934+0000 2023-11-25T05:04:42.245 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:42.243934+0000 2023-11-25T05:04:42.245 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:42.245 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:42.245 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:42.245 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:42.245 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:42.245 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:42.245 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:42.245 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:42.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.243+0000 7fc1b1b05640 1 Processor -- start 2023-11-25T05:04:42.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.243+0000 7fc1b1b05640 1 -- start start 2023-11-25T05:04:42.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.243+0000 7fc1b1b05640 10 monclient: init 2023-11-25T05:04:42.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.244+0000 7fc1b1b05640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:42.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.244+0000 7fc1b1b05640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-11-25T05:04:42.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.244+0000 7fc1b1b05640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1ac0afbf0 0x7fc1ac070610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:42.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.244+0000 7fc1b1b05640 10 monclient(hunting): picked mon.noname-b con 0x7fc1ac0afbf0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:42.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.244+0000 7fc1b1b05640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc1ac13b4a0 0x7fc1ac13b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:42.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.244+0000 7fc1b1b05640 10 monclient(hunting): picked mon.noname-c con 0x7fc1ac13b4a0 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:42.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.244+0000 7fc1b1b05640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc1ac071f10 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:42.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.244+0000 7fc1b1b05640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ac13be00 con 0x7fc1ac071f10 2023-11-25T05:04:42.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.244+0000 7fc1b1b05640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:42.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.244+0000 7fc1b1b05640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1ac072a70 con 0x7fc1ac13b4a0 2023-11-25T05:04:42.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.244+0000 7fc1b1b05640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:42.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.244+0000 7fc1b1b05640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac0affd0 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.244+0000 7fc1b1b05640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.244+0000 7fc1b1b05640 10 monclient(hunting): authenticate will time out at 1408.487297s 2023-11-25T05:04:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.244+0000 7fc1ab7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1ac0afbf0 0x7fc1ac070610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1abfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc1ac071f10 0x7fc1ac072360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56742/0 (socket says 172.21.15.7:56742) 2023-11-25T05:04:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1abfff640 1 -- 172.21.15.7:0/2079352041 learned_addr learned my addr 172.21.15.7:0/2079352041 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:04:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1aaffd640 1 --2- 172.21.15.7:0/2079352041 >> v2:172.21.15.90:3300/0 conn(0x7fc1ac13b4a0 0x7fc1ac13b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1ab7fe640 10 monclient(hunting): get_auth_request con 0x7fc1ac0afbf0 auth_method 0 2023-11-25T05:04:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1ab7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1ab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1aaffd640 10 monclient(hunting): get_auth_request con 0x7fc1ac13b4a0 auth_method 0 2023-11-25T05:04:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1aaffd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:42.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1aaffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:42.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:42.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:42.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:42.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:42.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.245+0000 7fc1aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:42.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.246+0000 7fc1aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:42.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.247+0000 7fc1ab7fe640 10 monclient(hunting): handle_auth_done global_id 4340 payload 274 2023-11-25T05:04:42.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.247+0000 7fc1ab7fe640 1 -- 172.21.15.7:0/2079352041 >> v2:172.21.15.90:3300/0 conn(0x7fc1ac13b4a0 msgr2=0x7fc1ac13b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:42.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.247+0000 7fc1ab7fe640 1 --2- 172.21.15.7:0/2079352041 >> v2:172.21.15.90:3300/0 conn(0x7fc1ac13b4a0 0x7fc1ac13b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.247+0000 7fc1ab7fe640 1 -- 172.21.15.7:0/2079352041 >> v1:172.21.15.7:6789/0 conn(0x7fc1ac071f10 legacy=0x7fc1ac072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:42.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.247+0000 7fc1ab7fe640 10 monclient: _finish_hunting 0 2023-11-25T05:04:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.247+0000 7fc1ab7fe640 1 monclient: found mon.noname-b 2023-11-25T05:04:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.247+0000 7fc1ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.247+0000 7fc1ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:04:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.247+0000 7fc1ab7fe640 1 -- 172.21.15.7:0/2079352041 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac06f2e0 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.247+0000 7fc1aaffd640 1 --2- 172.21.15.7:0/2079352041 >> v2:172.21.15.90:3300/0 conn(0x7fc1ac13b4a0 0x7fc1ac13b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-25T05:04:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.249+0000 7fc1ab7fe640 1 --2- 172.21.15.7:0/2079352041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1ac0afbf0 0x7fc1ac070610 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc1a00097a0 tx=0x7fc1a002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fcaeb0386ec6404 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:42.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1aa7fc640 1 -- 172.21.15.7:0/2079352041 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0031830 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:42.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:42.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1aa7fc640 10 monclient: dump: 2023-11-25T05:04:42.255 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:42.255 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:42.255 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.255 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.255 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:42.255 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:42.256 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:42.256 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:42.256 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:42.256 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:42.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1aa7fc640 10 monclient: _finish_auth 0 2023-11-25T05:04:42.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1aa7fc640 10 monclient: _check_auth_tickets 2023-11-25T05:04:42.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:42.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1aa7fc640 1 -- 172.21.15.7:0/2079352041 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a0031990 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1aa7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:42.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1aa7fc640 1 -- 172.21.15.7:0/2079352041 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0031d20 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:42.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:04:42.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1aa7fc640 10 monclient: dump: 2023-11-25T05:04:42.257 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:42.257 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:42.257 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.257 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.257 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:42.257 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:42.257 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:42.258 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:42.258 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:42.258 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:42.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1b1b05640 5 monclient: authenticate success, global_id 4340 2023-11-25T05:04:42.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1b1b05640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:42.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.253+0000 7fc1b1b05640 4 set_mon_vals no callback set 2023-11-25T05:04:42.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.254+0000 7fc1b1b05640 10 monclient: shutdown 2023-11-25T05:04:42.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.254+0000 7fc1b1b05640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:42.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.254+0000 7fc1b1b05640 1 -- 172.21.15.7:0/2079352041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1ac0afbf0 msgr2=0x7fc1ac070610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:42.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.254+0000 7fc1b1b05640 1 --2- 172.21.15.7:0/2079352041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1ac0afbf0 0x7fc1ac070610 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc1a00097a0 tx=0x7fc1a002b8d0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.254+0000 7fc1b1b05640 1 -- 172.21.15.7:0/2079352041 shutdown_connections 2023-11-25T05:04:42.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.254+0000 7fc1b1b05640 1 --2- 172.21.15.7:0/2079352041 >> v2:172.21.15.90:3300/0 conn(0x7fc1ac13b4a0 0x7fc1ac13b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.254+0000 7fc1b1b05640 1 --2- 172.21.15.7:0/2079352041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1ac0afbf0 0x7fc1ac070610 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.254+0000 7fc1b1b05640 1 -- 172.21.15.7:0/2079352041 >> 172.21.15.7:0/2079352041 conn(0x7fc1ac0b01e0 msgr2=0x7fc1ac0b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:42.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.254+0000 7fc1b1b05640 1 -- 172.21.15.7:0/2079352041 shutdown_connections 2023-11-25T05:04:42.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.254+0000 7fc1b1b05640 1 -- 172.21.15.7:0/2079352041 wait complete. 2023-11-25T05:04:42.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.254+0000 7fc1b1b05640 10 monclient: build_initial_monmap 2023-11-25T05:04:42.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.254+0000 7fc1b1b05640 10 monclient: monmap: 2023-11-25T05:04:42.265 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:42.265 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:42.265 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:42.265 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:42.265 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:42.265 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:42.265 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:42.266 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:42.266 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:42.266 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:42.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 1 librados: starting msgr at 2023-11-25T05:04:42.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 1 librados: starting objecter 2023-11-25T05:04:42.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 1 Processor -- start 2023-11-25T05:04:42.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 1 -- start start 2023-11-25T05:04:42.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 1 librados: setting wanted keys 2023-11-25T05:04:42.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 1 librados: calling monclient init 2023-11-25T05:04:42.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 10 monclient: init 2023-11-25T05:04:42.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:42.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-25T05:04:42.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1ac071f10 0x7fc1ac1325a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:42.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 10 monclient(hunting): picked mon.noname-a con 0x7fc1ac071f10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1ac0afbf0 0x7fc1ac132ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 10 monclient(hunting): picked mon.noname-b con 0x7fc1ac0afbf0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1ac13b4a0 0x7fc1ac1d5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 10 monclient(hunting): picked mon.noname-c con 0x7fc1ac13b4a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac1331b0 con 0x7fc1ac071f10 2023-11-25T05:04:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac134290 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac1346b0 con 0x7fc1ac13b4a0 2023-11-25T05:04:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.255+0000 7fc1b1b05640 10 monclient(hunting): authenticate will time out at 1408.498750s 2023-11-25T05:04:42.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1abfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1ac13b4a0 0x7fc1ac1d5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:42.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1aaffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1ac0afbf0 0x7fc1ac132ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:42.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1ab7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1ac071f10 0x7fc1ac1325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:42.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1abfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1ac13b4a0 0x7fc1ac1d5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:42044/0 (socket says 172.21.15.7:42044) 2023-11-25T05:04:42.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1abfff640 1 -- 172.21.15.7:0/731544009 learned_addr learned my addr 172.21.15.7:0/731544009 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:42.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1abfff640 10 monclient(hunting): get_auth_request con 0x7fc1ac13b4a0 auth_method 0 2023-11-25T05:04:42.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:42.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1abfff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:42.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1abfff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:42.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1aaffd640 10 monclient(hunting): get_auth_request con 0x7fc1ac0afbf0 auth_method 0 2023-11-25T05:04:42.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:42.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1aaffd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:42.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1aaffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:42.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1ab7fe640 10 monclient(hunting): get_auth_request con 0x7fc1ac071f10 auth_method 0 2023-11-25T05:04:42.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:42.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1ab7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:42.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.256+0000 7fc1ab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:42.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.257+0000 7fc1abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:42.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.257+0000 7fc1abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:42.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.257+0000 7fc1abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:42.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.257+0000 7fc1aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:42.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.257+0000 7fc1aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:42.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.257+0000 7fc1aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:42.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.257+0000 7fc1ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:42.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.257+0000 7fc1ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:42.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.257+0000 7fc1ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:42.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.259+0000 7fc1aaffd640 10 monclient(hunting): handle_auth_done global_id 4402 payload 824 2023-11-25T05:04:42.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.259+0000 7fc1aaffd640 1 -- 172.21.15.7:0/731544009 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1ac13b4a0 msgr2=0x7fc1ac1d5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:42.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.259+0000 7fc1aaffd640 1 --2- 172.21.15.7:0/731544009 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1ac13b4a0 0x7fc1ac1d5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.259+0000 7fc1aaffd640 1 -- 172.21.15.7:0/731544009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1ac071f10 msgr2=0x7fc1ac1325a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:42.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.259+0000 7fc1aaffd640 1 --2- 172.21.15.7:0/731544009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1ac071f10 0x7fc1ac1325a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.259+0000 7fc1aaffd640 10 monclient: _finish_hunting 0 2023-11-25T05:04:42.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.259+0000 7fc1aaffd640 1 monclient: found mon.noname-b 2023-11-25T05:04:42.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.259+0000 7fc1aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:42.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.259+0000 7fc1aaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:42.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.259+0000 7fc1aaffd640 1 -- 172.21.15.7:0/731544009 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c009400 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.262+0000 7fc1aaffd640 1 --2- 172.21.15.7:0/731544009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1ac0afbf0 0x7fc1ac132ae0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00e870 tx=0x7fc19c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:42.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.262+0000 7fc193fff640 10 client.?.objecter ms_handle_connect 0x7fc1ac0afbf0 2023-11-25T05:04:42.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.262+0000 7fc193fff640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:42.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc193fff640 1 -- 172.21.15.7:0/731544009 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c00c940 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc193fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:42.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc193fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:42.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc193fff640 10 monclient: dump: 2023-11-25T05:04:42.279 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:42.279 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:42.279 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.279 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.279 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:42.279 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:42.279 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:42.279 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:42.279 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:42.280 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:42.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc193fff640 10 monclient: _finish_auth 0 2023-11-25T05:04:42.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc193fff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:42.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc193fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:42.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc193fff640 1 -- 172.21.15.7:0/731544009 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19c00caa0 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc193fff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:42.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc193fff640 1 -- 172.21.15.7:0/731544009 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c00ce30 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc193fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:42.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc193fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:42.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc193fff640 10 monclient: dump: 2023-11-25T05:04:42.280 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:42.280 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:42.281 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc1b1b05640 5 monclient: authenticate success, global_id 4402 2023-11-25T05:04:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc1b1b05640 10 monclient: _renew_subs 2023-11-25T05:04:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc1b1b05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc1b1b05640 1 -- 172.21.15.7:0/731544009 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a0009400 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc1a9ffb640 4 set_mon_vals no callback set 2023-11-25T05:04:42.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc1b1b05640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:42.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc1b1b05640 10 monclient: _renew_subs 2023-11-25T05:04:42.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc1b1b05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:42.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc1b1b05640 1 -- 172.21.15.7:0/731544009 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ac1d5860 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.265+0000 7fc1b1b05640 1 librados: init done 2023-11-25T05:04:42.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.266+0000 7fc1b1b05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:42.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.266+0000 7fc1b1b05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:42.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.266+0000 7fc1b1b05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:42.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.266+0000 7fc1b1b05640 1 -- 172.21.15.7:0/731544009 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ac134100 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.286+0000 7fc193fff640 1 -- 172.21.15.7:0/731544009 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc19c013470 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.286+0000 7fc193fff640 1 --2- 172.21.15.7:0/731544009 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc19806c850 0x7fc19806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:42.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.287+0000 7fc1ab7fe640 1 --2- 172.21.15.7:0/731544009 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc19806c850 0x7fc19806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:42.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.287+0000 7fc1ab7fe640 10 monclient: get_auth_request con 0x7fc19806c850 auth_method 0 2023-11-25T05:04:42.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.287+0000 7fc1ab7fe640 1 --2- 172.21.15.7:0/731544009 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc19806c850 0x7fc19806ed10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc1a00097a0 tx=0x7fc1a003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:42.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.287+0000 7fc193fff640 10 client.4402.objecter ms_handle_connect 0x7fc19806c850 2023-11-25T05:04:42.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.289+0000 7fc193fff640 1 -- 172.21.15.7:0/731544009 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc19c058ca0 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.289+0000 7fc193fff640 10 client.4402.objecter ms_dispatch 0x7fc1ac000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:42.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.289+0000 7fc193fff640 3 client.4402.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:42.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.289+0000 7fc193fff640 3 client.4402.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:42.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.289+0000 7fc193fff640 20 client.4402.objecter dump_active .. 0 homeless 2023-11-25T05:04:42.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.527+0000 7fc193fff640 1 -- 172.21.15.7:0/731544009 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc19c054fe0 con 0x7fc1ac0afbf0 2023-11-25T05:04:42.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.527+0000 7fc193fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:42.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.527+0000 7fc193fff640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:42.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.707+0000 7fc1b1b05640 1 -- 172.21.15.7:0/731544009 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1ac0af0f0 con 0x7fc19806c850 2023-11-25T05:04:42.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.710+0000 7fc193fff640 1 -- 172.21.15.7:0/731544009 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48013 (secure 0 0 0) 0x7fc1a003a420 con 0x7fc19806c850 2023-11-25T05:04:42.712 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:04:42.712 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-25T05:04:42.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 10 librados: watch_flush enter 2023-11-25T05:04:42.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 10 librados: watch_flush exit 2023-11-25T05:04:42.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 20 client.4402.objecter shutdown clearing up homeless session... 2023-11-25T05:04:42.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 10 client.4402.objecter successfully canceled tick 2023-11-25T05:04:42.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 1 -- 172.21.15.7:0/731544009 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc19806c850 msgr2=0x7fc19806ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:42.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 1 --2- 172.21.15.7:0/731544009 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc19806c850 0x7fc19806ed10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc1a00097a0 tx=0x7fc1a003c040 comp rx=0 tx=0).stop 2023-11-25T05:04:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 10 monclient: shutdown 2023-11-25T05:04:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 1 -- 172.21.15.7:0/731544009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1ac0afbf0 msgr2=0x7fc1ac132ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 1 --2- 172.21.15.7:0/731544009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1ac0afbf0 0x7fc1ac132ae0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00e870 tx=0x7fc19c00ed30 comp rx=0 tx=0).stop 2023-11-25T05:04:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 1 -- 172.21.15.7:0/731544009 shutdown_connections 2023-11-25T05:04:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 1 --2- 172.21.15.7:0/731544009 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc19806c850 0x7fc19806ed10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 1 --2- 172.21.15.7:0/731544009 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1ac13b4a0 0x7fc1ac1d5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 1 --2- 172.21.15.7:0/731544009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1ac0afbf0 0x7fc1ac132ae0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 1 --2- 172.21.15.7:0/731544009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1ac071f10 0x7fc1ac1325a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 1 -- 172.21.15.7:0/731544009 >> 172.21.15.7:0/731544009 conn(0x7fc1ac0b01e0 msgr2=0x7fc1ac0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 1 -- 172.21.15.7:0/731544009 shutdown_connections 2023-11-25T05:04:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 1 -- 172.21.15.7:0/731544009 wait complete. 2023-11-25T05:04:42.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.712+0000 7fc1b1b05640 1 librados: shutdown 2023-11-25T05:04:42.727 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-11-25T05:04:41.373571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218832,"kb_used_data":2468,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755888,"statfs":{"total":773094113280,"available":772870029312,"internally_reserved":0,"allocated":2527232,"data_stored":1395084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":58,"apply_latency_ms":58,"commit_latency_ns":58000000,"apply_latency_ns":58000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001536"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:36.798887+0000","last_change":"2023-11-25T05:04:33.582978+0000","last_active":"2023-11-25T05:04:36.798887+0000","last_peered":"2023-11-25T05:04:36.798887+0000","last_clean":"2023-11-25T05:04:36.798887+0000","last_became_active":"2023-11-25T05:04:33.577510+0000","last_became_peered":"2023-11-25T05:04:33.577510+0000","last_unstale":"2023-11-25T05:04:36.798887+0000","last_undegraded":"2023-11-25T05:04:36.798887+0000","last_fullsized":"2023-11-25T05:04:36.798887+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:53:20.091826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:36.765181+0000","last_change":"2023-11-25T05:04:33.645680+0000","last_active":"2023-11-25T05:04:36.765181+0000","last_peered":"2023-11-25T05:04:36.765181+0000","last_clean":"2023-11-25T05:04:36.765181+0000","last_became_active":"2023-11-25T05:04:33.610776+0000","last_became_peered":"2023-11-25T05:04:33.610776+0000","last_unstale":"2023-11-25T05:04:36.765181+0000","last_undegraded":"2023-11-25T05:04:36.765181+0000","last_fullsized":"2023-11-25T05:04:36.765181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:02:08.013900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:36.789348+0000","last_change":"2023-11-25T05:04:33.350255+0000","last_active":"2023-11-25T05:04:36.789348+0000","last_peered":"2023-11-25T05:04:36.789348+0000","last_clean":"2023-11-25T05:04:36.789348+0000","last_became_active":"2023-11-25T05:04:33.337158+0000","last_became_peered":"2023-11-25T05:04:33.337158+0000","last_unstale":"2023-11-25T05:04:36.789348+0000","last_undegraded":"2023-11-25T05:04:36.789348+0000","last_fullsized":"2023-11-25T05:04:36.789348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:25:34.365788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:35.704622+0000","last_change":"2023-11-25T05:04:33.396056+0000","last_active":"2023-11-25T05:04:35.704622+0000","last_peered":"2023-11-25T05:04:35.704622+0000","last_clean":"2023-11-25T05:04:35.704622+0000","last_became_active":"2023-11-25T05:04:33.388849+0000","last_became_peered":"2023-11-25T05:04:33.388849+0000","last_unstale":"2023-11-25T05:04:35.704622+0000","last_undegraded":"2023-11-25T05:04:35.704622+0000","last_fullsized":"2023-11-25T05:04:35.704622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:11:08.763309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:36.762061+0000","last_change":"2023-11-25T05:04:33.381810+0000","last_active":"2023-11-25T05:04:36.762061+0000","last_peered":"2023-11-25T05:04:36.762061+0000","last_clean":"2023-11-25T05:04:36.762061+0000","last_became_active":"2023-11-25T05:04:33.376375+0000","last_became_peered":"2023-11-25T05:04:33.376375+0000","last_unstale":"2023-11-25T05:04:36.762061+0000","last_undegraded":"2023-11-25T05:04:36.762061+0000","last_fullsized":"2023-11-25T05:04:36.762061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:41:46.038487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-25T05:04:34.697761+0000","last_change":"2023-11-25T05:04:33.238092+0000","last_active":"2023-11-25T05:04:34.697761+0000","last_peered":"2023-11-25T05:04:34.697761+0000","last_clean":"2023-11-25T05:04:34.697761+0000","last_became_active":"2023-11-25T05:04:33.196533+0000","last_became_peered":"2023-11-25T05:04:33.196533+0000","last_unstale":"2023-11-25T05:04:34.697761+0000","last_undegraded":"2023-11-25T05:04:34.697761+0000","last_fullsized":"2023-11-25T05:04:34.697761+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:15:44.127680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:36.775112+0000","last_change":"2023-11-25T05:04:33.294027+0000","last_active":"2023-11-25T05:04:36.775112+0000","last_peered":"2023-11-25T05:04:36.775112+0000","last_clean":"2023-11-25T05:04:36.775112+0000","last_became_active":"2023-11-25T05:04:33.242068+0000","last_became_peered":"2023-11-25T05:04:33.242068+0000","last_unstale":"2023-11-25T05:04:36.775112+0000","last_undegraded":"2023-11-25T05:04:36.775112+0000","last_fullsized":"2023-11-25T05:04:36.775112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:37:54.756624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:36.784564+0000","last_change":"2023-11-25T05:04:33.784328+0000","last_active":"2023-11-25T05:04:36.784564+0000","last_peered":"2023-11-25T05:04:36.784564+0000","last_clean":"2023-11-25T05:04:36.784564+0000","last_became_active":"2023-11-25T05:04:33.778042+0000","last_became_peered":"2023-11-25T05:04:33.778042+0000","last_unstale":"2023-11-25T05:04:36.784564+0000","last_undegraded":"2023-11-25T05:04:36.784564+0000","last_fullsized":"2023-11-25T05:04:36.784564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:40:38.193703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":106,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-25T05:04:37.659883+0000","last_change":"2023-11-25T05:04:28.241367+0000","last_active":"2023-11-25T05:04:37.659883+0000","last_peered":"2023-11-25T05:04:37.659883+0000","last_clean":"2023-11-25T05:04:37.659883+0000","last_became_active":"2023-11-25T05:04:27.388156+0000","last_became_peered":"2023-11-25T05:04:27.388156+0000","last_unstale":"2023-11-25T05:04:37.659883+0000","last_undegraded":"2023-11-25T05:04:37.659883+0000","last_fullsized":"2023-11-25T05:04:37.659883+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:22.310975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:22.310975+0000","last_clean_scrub_stamp":"2023-11-25T05:04:22.310975+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:50:37.122665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":16,"seq":68719476742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8249999999999993}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.350000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.161999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.564}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.702}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.239000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.321999999999999}]}]},{"osd":1,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":618496,"data_stored":473484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.693}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.5069999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.724}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.3490000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.895}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.904}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.871}]}]},{"osd":3,"up_from":16,"seq":68719476742,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.898}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.305}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.527000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4280000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.109000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.214}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.658999999999999}]}]},{"osd":0,"up_from":16,"seq":68719476742,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9130000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.903}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.349}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.143999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.132}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.686}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.765999999999998}]}]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.373000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6449999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.619999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.332000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.343999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.07}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.684999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9510000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4559999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.611000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.698}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.23}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.571000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.534000000000001}]}]},{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":69896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":142.904}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.012}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.472000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":172.67400000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.224}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.960000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.268000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":618496,"data_stored":473484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8810000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.981}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.585000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.914000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.41}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.097999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.419}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-25T05:04:42.729 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-25T05:04:42.729 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-25T05:04:42.729 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-25T05:04:42.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-25T05:04:42.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.836+0000 7fb02d875640 10 monclient: get_monmap_and_config 2023-11-25T05:04:42.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.836+0000 7fb02d875640 10 monclient: build_initial_monmap 2023-11-25T05:04:42.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.836+0000 7fb02d875640 10 monclient: monmap: 2023-11-25T05:04:42.838 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:42.838 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:42.838 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:04:42.837183+0000 2023-11-25T05:04:42.838 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:04:42.837183+0000 2023-11-25T05:04:42.838 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:42.839 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:42.839 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:04:42.839 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:04:42.839 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:04:42.839 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:04:42.839 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:04:42.839 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:42.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.836+0000 7fb02d875640 1 Processor -- start 2023-11-25T05:04:42.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.836+0000 7fb02d875640 1 -- start start 2023-11-25T05:04:42.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.836+0000 7fb02d875640 10 monclient: init 2023-11-25T05:04:42.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.837+0000 7fb02d875640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.837+0000 7fb02d875640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-11-25T05:04:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.837+0000 7fb02d875640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0280a8a30 0x7fb0281424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.837+0000 7fb02d875640 10 monclient(hunting): picked mon.noname-a con 0x7fb0280a8a30 addr v2:172.21.15.7:3300/0 2023-11-25T05:04:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.837+0000 7fb02d875640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb0280a8e10 addr v1:172.21.15.7:6789/0 2023-11-25T05:04:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.837+0000 7fb02d875640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb028142a20 0x7fb028145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.837+0000 7fb02d875640 10 monclient(hunting): picked mon.noname-c con 0x7fb028142a20 addr v2:172.21.15.90:3300/0 2023-11-25T05:04:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.837+0000 7fb02d875640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb028142e70 con 0x7fb0280a8e10 2023-11-25T05:04:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.837+0000 7fb02d875640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.837+0000 7fb02d875640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0280aaa10 con 0x7fb0280a8a30 2023-11-25T05:04:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.837+0000 7fb02d875640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.837+0000 7fb02d875640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0280aac10 con 0x7fb028142a20 2023-11-25T05:04:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.837+0000 7fb02d875640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.837+0000 7fb02d875640 10 monclient(hunting): authenticate will time out at 1409.080458s 2023-11-25T05:04:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.838+0000 7fb026ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0280a8a30 0x7fb0281424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.838+0000 7fb0277fe640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb028142a20 0x7fb028145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.838+0000 7fb0277fe640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb028142a20 0x7fb028145bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:35420/0 (socket says 172.21.15.7:35420) 2023-11-25T05:04:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.838+0000 7fb0277fe640 1 -- 172.21.15.7:0/3394926875 learned_addr learned my addr 172.21.15.7:0/3394926875 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.838+0000 7fb0277fe640 10 monclient(hunting): get_auth_request con 0x7fb028142a20 auth_method 0 2023-11-25T05:04:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.838+0000 7fb0277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.838+0000 7fb0277fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.838+0000 7fb0277fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.838+0000 7fb026ffd640 10 monclient(hunting): get_auth_request con 0x7fb0280a8a30 auth_method 0 2023-11-25T05:04:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.838+0000 7fb026ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.838+0000 7fb026ffd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.838+0000 7fb026ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.839+0000 7fb0277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.839+0000 7fb0277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.839+0000 7fb0277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.839+0000 7fb026ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.839+0000 7fb026ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.839+0000 7fb026ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:42.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.840+0000 7fb0277fe640 10 monclient(hunting): handle_auth_done global_id 4405 payload 274 2023-11-25T05:04:42.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.840+0000 7fb0277fe640 1 -- 172.21.15.7:0/3394926875 >> v2:172.21.15.7:3300/0 conn(0x7fb0280a8a30 msgr2=0x7fb0281424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:42.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.840+0000 7fb0277fe640 1 --2- 172.21.15.7:0/3394926875 >> v2:172.21.15.7:3300/0 conn(0x7fb0280a8a30 0x7fb0281424e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.840+0000 7fb0277fe640 1 -- 172.21.15.7:0/3394926875 >> v1:172.21.15.7:6789/0 conn(0x7fb0280a8e10 legacy=0x7fb0281454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:04:42.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.840+0000 7fb0277fe640 10 monclient: _finish_hunting 0 2023-11-25T05:04:42.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.840+0000 7fb0277fe640 1 monclient: found mon.noname-c 2023-11-25T05:04:42.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.840+0000 7fb0277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:42.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.840+0000 7fb0277fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:04:42.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.840+0000 7fb0277fe640 1 -- 172.21.15.7:0/3394926875 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0280aaef0 con 0x7fb028142a20 2023-11-25T05:04:42.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.842+0000 7fb0277fe640 1 --2- 172.21.15.7:0/3394926875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb028142a20 0x7fb028145bb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb0280b3a60 tx=0x7fb01802bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b81f64fb49ce377 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:42.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb025ffb640 1 -- 172.21.15.7:0/3394926875 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01802cb20 con 0x7fb028142a20 2023-11-25T05:04:42.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb025ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:42.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb025ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:04:42.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb025ffb640 10 monclient: dump: 2023-11-25T05:04:42.848 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:42.850 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb025ffb640 10 monclient: _finish_auth 0 2023-11-25T05:04:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb025ffb640 10 monclient: _check_auth_tickets 2023-11-25T05:04:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb025ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb025ffb640 1 -- 172.21.15.7:0/3394926875 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01802cc80 con 0x7fb028142a20 2023-11-25T05:04:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb025ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb025ffb640 1 -- 172.21.15.7:0/3394926875 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0180343f0 con 0x7fb028142a20 2023-11-25T05:04:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb025ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb025ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb025ffb640 10 monclient: dump: 2023-11-25T05:04:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:42.852 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:42.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb02d875640 5 monclient: authenticate success, global_id 4405 2023-11-25T05:04:42.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb02d875640 10 monclient: get_monmap_and_config success 2023-11-25T05:04:42.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb02d875640 4 set_mon_vals no callback set 2023-11-25T05:04:42.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb02d875640 10 monclient: shutdown 2023-11-25T05:04:42.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb02d875640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:42.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb02d875640 1 -- 172.21.15.7:0/3394926875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb028142a20 msgr2=0x7fb028145bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:42.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb02d875640 1 --2- 172.21.15.7:0/3394926875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb028142a20 0x7fb028145bb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb0280b3a60 tx=0x7fb01802bf80 comp rx=0 tx=0).stop 2023-11-25T05:04:42.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb02d875640 1 -- 172.21.15.7:0/3394926875 shutdown_connections 2023-11-25T05:04:42.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb02d875640 1 --2- 172.21.15.7:0/3394926875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb028142a20 0x7fb028145bb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.846+0000 7fb02d875640 1 --2- 172.21.15.7:0/3394926875 >> v2:172.21.15.7:3300/0 conn(0x7fb0280a8a30 0x7fb0281424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.847+0000 7fb02d875640 1 -- 172.21.15.7:0/3394926875 >> 172.21.15.7:0/3394926875 conn(0x7fb028061a30 msgr2=0x7fb0280b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:42.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.847+0000 7fb02d875640 1 -- 172.21.15.7:0/3394926875 shutdown_connections 2023-11-25T05:04:42.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.847+0000 7fb02d875640 1 -- 172.21.15.7:0/3394926875 wait complete. 2023-11-25T05:04:42.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.847+0000 7fb02d875640 10 monclient: build_initial_monmap 2023-11-25T05:04:42.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.847+0000 7fb02d875640 10 monclient: monmap: 2023-11-25T05:04:42.857 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:04:42.857 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:04:42.857 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:04:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:04:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:04:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:04:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:04:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:04:42.858 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.847+0000 7fb02d875640 1 librados: starting msgr at 2023-11-25T05:04:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.847+0000 7fb02d875640 1 librados: starting objecter 2023-11-25T05:04:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 1 Processor -- start 2023-11-25T05:04:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 1 -- start start 2023-11-25T05:04:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 1 librados: setting wanted keys 2023-11-25T05:04:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 1 librados: calling monclient init 2023-11-25T05:04:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 10 monclient: init 2023-11-25T05:04:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 10 monclient: _reopen_session rank -1 2023-11-25T05:04:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:04:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0280a8a30 0x7fb0281da8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 10 monclient(hunting): picked mon.noname-a con 0x7fb0280a8a30 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:04:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0280a8e10 0x7fb02813b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 10 monclient(hunting): picked mon.noname-c con 0x7fb0280a8e10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:04:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb028142a20 0x7fb02813bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 10 monclient(hunting): picked mon.noname-b con 0x7fb028142a20 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:04:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02813c320 con 0x7fb0280a8a30 2023-11-25T05:04:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0280adfb0 con 0x7fb028142a20 2023-11-25T05:04:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 10 monclient(hunting): start opening mon connection 2023-11-25T05:04:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0280aeb80 con 0x7fb0280a8e10 2023-11-25T05:04:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 10 monclient(hunting): _renew_subs 2023-11-25T05:04:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.848+0000 7fb02d875640 10 monclient(hunting): authenticate will time out at 1409.091719s 2023-11-25T05:04:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb0267fc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0280a8e10 0x7fb02813b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb0277fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb028142a20 0x7fb02813bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb026ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0280a8a30 0x7fb0281da8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb0267fc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0280a8e10 0x7fb02813b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:42046/0 (socket says 172.21.15.7:42046) 2023-11-25T05:04:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb0267fc640 1 -- 172.21.15.7:0/4082997244 learned_addr learned my addr 172.21.15.7:0/4082997244 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:04:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb0267fc640 10 monclient(hunting): get_auth_request con 0x7fb0280a8e10 auth_method 0 2023-11-25T05:04:42.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb0267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:42.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb0267fc640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:42.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb0267fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:42.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb026ffd640 10 monclient(hunting): get_auth_request con 0x7fb0280a8a30 auth_method 0 2023-11-25T05:04:42.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb026ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:42.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb026ffd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:42.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb026ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:42.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb0277fe640 10 monclient(hunting): get_auth_request con 0x7fb028142a20 auth_method 0 2023-11-25T05:04:42.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb0277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:04:42.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb0277fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:04:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.849+0000 7fb0277fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:04:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.850+0000 7fb0267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.850+0000 7fb0267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.850+0000 7fb0267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.850+0000 7fb0277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.850+0000 7fb0277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.850+0000 7fb0277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.850+0000 7fb026ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:04:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.850+0000 7fb026ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:04:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.850+0000 7fb026ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:04:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.852+0000 7fb0277fe640 10 monclient(hunting): handle_auth_done global_id 4408 payload 824 2023-11-25T05:04:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.852+0000 7fb0277fe640 1 -- 172.21.15.7:0/4082997244 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0280a8e10 msgr2=0x7fb02813b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.852+0000 7fb0277fe640 1 --2- 172.21.15.7:0/4082997244 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0280a8e10 0x7fb02813b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.852+0000 7fb0277fe640 1 -- 172.21.15.7:0/4082997244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0280a8a30 msgr2=0x7fb0281da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.852+0000 7fb0277fe640 1 --2- 172.21.15.7:0/4082997244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0280a8a30 0x7fb0281da8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.852+0000 7fb0277fe640 10 monclient: _finish_hunting 0 2023-11-25T05:04:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.852+0000 7fb0277fe640 1 monclient: found mon.noname-b 2023-11-25T05:04:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.852+0000 7fb0277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:04:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.852+0000 7fb0277fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:04:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.852+0000 7fb0277fe640 1 -- 172.21.15.7:0/4082997244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb018009b30 con 0x7fb028142a20 2023-11-25T05:04:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.855+0000 7fb0277fe640 1 --2- 172.21.15.7:0/4082997244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb028142a20 0x7fb02813bde0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb018032000 tx=0x7fb018004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:42.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.855+0000 7fb01ffff640 10 client.?.objecter ms_handle_connect 0x7fb028142a20 2023-11-25T05:04:42.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.855+0000 7fb01ffff640 10 client.?.objecter resend_mon_ops 2023-11-25T05:04:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb01ffff640 1 -- 172.21.15.7:0/4082997244 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb018009510 con 0x7fb028142a20 2023-11-25T05:04:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb01ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb01ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:04:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb01ffff640 10 monclient: dump: 2023-11-25T05:04:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:42.867 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:42.867 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:42.867 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:42.867 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:42.867 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:42.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb01ffff640 10 monclient: _finish_auth 0 2023-11-25T05:04:42.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb01ffff640 10 monclient: _check_auth_tickets 2023-11-25T05:04:42.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb01ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:04:42.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb01ffff640 1 -- 172.21.15.7:0/4082997244 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb018007e00 con 0x7fb028142a20 2023-11-25T05:04:42.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb01ffff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:04:42.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb01ffff640 1 -- 172.21.15.7:0/4082997244 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01803e690 con 0x7fb028142a20 2023-11-25T05:04:42.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb01ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:04:42.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb01ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:04:42.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb01ffff640 10 monclient: dump: 2023-11-25T05:04:42.868 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:04:42.868 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:04:42.868 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.868 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:04:42.868 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:04:42.868 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:04:42.868 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:04:42.869 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:04:42.869 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:04:42.869 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:04:42.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb02d875640 5 monclient: authenticate success, global_id 4408 2023-11-25T05:04:42.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb02d875640 10 monclient: _renew_subs 2023-11-25T05:04:42.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb02d875640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:42.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb02d875640 1 -- 172.21.15.7:0/4082997244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0280ac2c0 con 0x7fb028142a20 2023-11-25T05:04:42.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb0257fa640 4 set_mon_vals no callback set 2023-11-25T05:04:42.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb02d875640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:04:42.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb02d875640 10 monclient: _renew_subs 2023-11-25T05:04:42.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb02d875640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:42.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb02d875640 1 -- 172.21.15.7:0/4082997244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0281db330 con 0x7fb028142a20 2023-11-25T05:04:42.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.858+0000 7fb02d875640 1 librados: init done 2023-11-25T05:04:42.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.859+0000 7fb02d875640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:42.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.859+0000 7fb02d875640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:42.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.859+0000 7fb02d875640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:42.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:42.859+0000 7fb02d875640 1 -- 172.21.15.7:0/4082997244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb028067ac0 con 0x7fb028142a20 2023-11-25T05:04:43.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.100+0000 7fb01ffff640 1 -- 172.21.15.7:0/4082997244 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb01803e7f0 con 0x7fb028142a20 2023-11-25T05:04:43.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.101+0000 7fb01ffff640 1 --2- 172.21.15.7:0/4082997244 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb00806c820 0x7fb00806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:04:43.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.101+0000 7fb026ffd640 1 --2- 172.21.15.7:0/4082997244 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb00806c820 0x7fb00806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:04:43.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.101+0000 7fb026ffd640 10 monclient: get_auth_request con 0x7fb00806c820 auth_method 0 2023-11-25T05:04:43.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.101+0000 7fb026ffd640 1 --2- 172.21.15.7:0/4082997244 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb00806c820 0x7fb00806ece0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb020009b40 tx=0x7fb020009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:04:43.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.101+0000 7fb01ffff640 10 client.4408.objecter ms_handle_connect 0x7fb00806c820 2023-11-25T05:04:43.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.103+0000 7fb01ffff640 1 -- 172.21.15.7:0/4082997244 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb01807b2d0 con 0x7fb028142a20 2023-11-25T05:04:43.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.103+0000 7fb01ffff640 10 client.4408.objecter ms_dispatch 0x7fb028000b90 osd_map(22..22 src has 1..22) v4 2023-11-25T05:04:43.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.103+0000 7fb01ffff640 3 client.4408.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:04:43.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.103+0000 7fb01ffff640 3 client.4408.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:04:43.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.103+0000 7fb01ffff640 20 client.4408.objecter dump_active .. 0 homeless 2023-11-25T05:04:43.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.113+0000 7fb01ffff640 1 -- 172.21.15.7:0/4082997244 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb018077610 con 0x7fb028142a20 2023-11-25T05:04:43.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.113+0000 7fb01ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:04:43.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.113+0000 7fb01ffff640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:04:43.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.333+0000 7fb02d875640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-11-25T05:04:43.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.333+0000 7fb02d875640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-11-25T05:04:43.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.333+0000 7fb02d875640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:04:43.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.333+0000 7fb02d875640 1 -- 172.21.15.7:0/4082997244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb028075b60 con 0x7fb028142a20 2023-11-25T05:04:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.340+0000 7fb01ffff640 1 -- 172.21.15.7:0/4082997244 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fb018043040 con 0x7fb028142a20 2023-11-25T05:04:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.340+0000 7fb01ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-11-25T05:04:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.340+0000 7fb01ffff640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:04:43.343 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:04:43.343 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-25T05:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.342+0000 7fb02d875640 10 librados: watch_flush enter 2023-11-25T05:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.342+0000 7fb02d875640 10 librados: watch_flush exit 2023-11-25T05:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.342+0000 7fb02d875640 20 client.4408.objecter shutdown clearing up homeless session... 2023-11-25T05:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.342+0000 7fb02d875640 10 client.4408.objecter successfully canceled tick 2023-11-25T05:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.343+0000 7fb02d875640 1 -- 172.21.15.7:0/4082997244 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb00806c820 msgr2=0x7fb00806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.343+0000 7fb02d875640 1 --2- 172.21.15.7:0/4082997244 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb00806c820 0x7fb00806ece0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb020009b40 tx=0x7fb020009190 comp rx=0 tx=0).stop 2023-11-25T05:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.343+0000 7fb02d875640 10 monclient: shutdown 2023-11-25T05:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.343+0000 7fb02d875640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.343+0000 7fb02d875640 1 -- 172.21.15.7:0/4082997244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb028142a20 msgr2=0x7fb02813bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.343+0000 7fb02d875640 1 --2- 172.21.15.7:0/4082997244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb028142a20 0x7fb02813bde0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb018032000 tx=0x7fb018004a40 comp rx=0 tx=0).stop 2023-11-25T05:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.343+0000 7fb02d875640 1 -- 172.21.15.7:0/4082997244 shutdown_connections 2023-11-25T05:04:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.343+0000 7fb02d875640 1 --2- 172.21.15.7:0/4082997244 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb00806c820 0x7fb00806ece0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.343+0000 7fb02d875640 1 --2- 172.21.15.7:0/4082997244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb028142a20 0x7fb02813bde0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.343+0000 7fb02d875640 1 --2- 172.21.15.7:0/4082997244 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0280a8e10 0x7fb02813b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.343+0000 7fb02d875640 1 --2- 172.21.15.7:0/4082997244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0280a8a30 0x7fb0281da8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:04:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.343+0000 7fb02d875640 1 -- 172.21.15.7:0/4082997244 >> 172.21.15.7:0/4082997244 conn(0x7fb028061a30 msgr2=0x7fb0280b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:04:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.343+0000 7fb02d875640 1 -- 172.21.15.7:0/4082997244 shutdown_connections 2023-11-25T05:04:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.343+0000 7fb02d875640 1 -- 172.21.15.7:0/4082997244 wait complete. 2023-11-25T05:04:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:04:43.343+0000 7fb02d875640 1 librados: shutdown 2023-11-25T05:04:43.356 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-25T05:04:43.356 INFO:teuthology.run_tasks:Running task workunit... 2023-11-25T05:04:43.369 INFO:tasks.workunit:Pulling workunits from ref ec928368edcc5fc71884a56fd9fd821b148f4e30 2023-11-25T05:04:43.369 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-11-25T05:04:43.370 DEBUG:teuthology.orchestra.run.smithi007:> stat -- /home/ubuntu/cephtest/mnt.0 2023-11-25T05:04:43.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-25T05:04:43.387 INFO:teuthology.orchestra.run.smithi007.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-11-25T05:04:43.387 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-11-25T05:04:43.443 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-11-25T05:04:43.443 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-11-25T05:04:43.499 INFO:tasks.workunit:timeout=6h 2023-11-25T05:04:43.499 INFO:tasks.workunit:cleanup=True 2023-11-25T05:04:43.499 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ec928368edcc5fc71884a56fd9fd821b148f4e30 2023-11-25T05:04:43.558 INFO:tasks.workunit.client.0.smithi007.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-11-25T05:04:53.083 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:35.281 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:53.086 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:35.284 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:53.129 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:35.272 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:53.132 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:35.276 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:53.136 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:35.334 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:53.140 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:35.338 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:53.163 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:35.361 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:53.168 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:35.312 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:04:53.184 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:35.382 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:13.093 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:55.290 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:13.176 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:55.374 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:13.201 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:55.399 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:13.222 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:55.366 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:13.223 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:55.367 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:13.242 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:55.383 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:13.256 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:55.454 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:13.261 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:55.459 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:13.288 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:55.486 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:33.206 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:15.404 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:33.210 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:15.408 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:33.223 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:15.422 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:33.252 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:15.396 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:33.253 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:15.398 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:33.262 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:15.460 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:33.263 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:15.462 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:33.267 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:15.411 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:33.277 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:15.475 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:33.601 INFO:tasks.workunit.client.0.smithi007.stderr:Updating files: 99% (11552/11620) Updating files: 100% (11620/11620) Updating files: 100% (11620/11620), done. 2023-11-25T05:05:34.492 INFO:tasks.workunit.client.0.smithi007.stderr:Note: switching to 'ec928368edcc5fc71884a56fd9fd821b148f4e30'. 2023-11-25T05:05:34.492 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-11-25T05:05:34.492 INFO:tasks.workunit.client.0.smithi007.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-11-25T05:05:34.492 INFO:tasks.workunit.client.0.smithi007.stderr:changes and commit them, and you can discard any commits you make in this 2023-11-25T05:05:34.492 INFO:tasks.workunit.client.0.smithi007.stderr:state without impacting any branches by switching back to a branch. 2023-11-25T05:05:34.492 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-11-25T05:05:34.492 INFO:tasks.workunit.client.0.smithi007.stderr:If you want to create a new branch to retain commits you create, you may 2023-11-25T05:05:34.493 INFO:tasks.workunit.client.0.smithi007.stderr:do so (now or later) by using -c with the switch command. Example: 2023-11-25T05:05:34.493 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-11-25T05:05:34.493 INFO:tasks.workunit.client.0.smithi007.stderr: git switch -c 2023-11-25T05:05:34.493 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-11-25T05:05:34.493 INFO:tasks.workunit.client.0.smithi007.stderr:Or undo this operation with: 2023-11-25T05:05:34.493 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-11-25T05:05:34.493 INFO:tasks.workunit.client.0.smithi007.stderr: git switch - 2023-11-25T05:05:34.493 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-11-25T05:05:34.493 INFO:tasks.workunit.client.0.smithi007.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-11-25T05:05:34.493 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-11-25T05:05:34.493 INFO:tasks.workunit.client.0.smithi007.stderr:HEAD is now at ec928368edc Merge pull request #52931 from yuvalif/wip-yuval-lua-teuthology 2023-11-25T05:05:34.497 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-11-25T05:05:34.559 INFO:tasks.workunit.client.0.smithi007.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-11-25T05:05:34.561 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-25T05:05:34.561 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-11-25T05:05:35.230 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-11-25T05:05:35.296 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-11-25T05:05:35.337 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-25T05:05:35.339 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-25T05:05:35.339 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-11-25T05:05:35.381 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-25T05:05:35.385 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-25T05:05:35.385 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-11-25T05:05:35.441 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2023-11-25T05:05:35.443 INFO:tasks.workunit:Running workunit rados/test.sh... 2023-11-25T05:05:35.443 DEBUG:teuthology.orchestra.run.smithi007:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ec928368edcc5fc71884a56fd9fd821b148f4e30 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2023-11-25T05:05:35.503 INFO:tasks.workunit.client.0.smithi007.stderr:+ parallel=1 2023-11-25T05:05:35.504 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' '' = --serial ']' 2023-11-25T05:05:35.504 INFO:tasks.workunit.client.0.smithi007.stderr:+ color= 2023-11-25T05:05:35.504 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -t 1 ']' 2023-11-25T05:05:35.504 INFO:tasks.workunit.client.0.smithi007.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2023-11-25T05:05:35.504 INFO:tasks.workunit.client.0.smithi007.stderr:+ declare -A pids 2023-11-25T05:05:35.504 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.504 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.504 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_aio 2023-11-25T05:05:35.504 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_aio' 2023-11-25T05:05:35.504 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_aio 2023-11-25T05:05:35.504 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.505 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_aio 2023-11-25T05:05:35.505 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92624 2023-11-25T05:05:35.505 INFO:tasks.workunit.client.0.smithi007.stdout:test api_aio on pid 92624 2023-11-25T05:05:35.509 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_aio on pid 92624' 2023-11-25T05:05:35.509 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=92624 2023-11-25T05:05:35.509 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.509 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.509 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2023-11-25T05:05:35.510 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_aio_pp 2023-11-25T05:05:35.510 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_aio_pp' 2023-11-25T05:05:35.510 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_aio_pp 2023-11-25T05:05:35.510 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.510 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.510 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.510 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.510 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.510 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.510 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.511 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.511 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.512 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.512 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.512 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.512 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.512 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.512 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.512 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.512 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.512 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_aio_pp 2023-11-25T05:05:35.512 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92631 2023-11-25T05:05:35.513 INFO:tasks.workunit.client.0.smithi007.stdout:test api_aio_pp on pid 92631 2023-11-25T05:05:35.515 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_aio_pp on pid 92631' 2023-11-25T05:05:35.516 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=92631 2023-11-25T05:05:35.516 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.516 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2023-11-25T05:05:35.516 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.516 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_io 2023-11-25T05:05:35.516 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_io' 2023-11-25T05:05:35.516 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_io 2023-11-25T05:05:35.516 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.516 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.516 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.516 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.517 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.517 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.517 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.517 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.517 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.517 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.517 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.517 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.517 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.517 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.517 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.517 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.517 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.518 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.518 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.518 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.518 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.518 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.518 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.518 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.518 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.518 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.518 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.518 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.518 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.519 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.519 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.519 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.519 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.519 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.519 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.519 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.519 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.519 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.519 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.519 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.519 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.519 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.520 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.520 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.520 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.520 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.520 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_io 2023-11-25T05:05:35.520 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92640 2023-11-25T05:05:35.520 INFO:tasks.workunit.client.0.smithi007.stdout:test api_io on pid 92640 2023-11-25T05:05:35.524 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_io on pid 92640' 2023-11-25T05:05:35.524 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=92640 2023-11-25T05:05:35.524 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.524 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.524 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2023-11-25T05:05:35.524 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_io_pp 2023-11-25T05:05:35.524 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_io_pp' 2023-11-25T05:05:35.525 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_io_pp 2023-11-25T05:05:35.525 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.525 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.525 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.525 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.525 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.525 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.525 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.525 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.525 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.525 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.526 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.526 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.526 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.526 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.526 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.526 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.526 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.526 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.526 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.526 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.526 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.526 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.527 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.527 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.527 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.527 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.527 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.527 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.527 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.527 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.527 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.527 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.527 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.527 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.528 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.528 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.528 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.528 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.528 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.528 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.528 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.528 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.528 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.528 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.528 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.528 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.528 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.529 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.529 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.529 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.529 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.529 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.529 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.529 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.529 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.529 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_io_pp 2023-11-25T05:05:35.529 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92653 2023-11-25T05:05:35.535 INFO:tasks.workunit.client.0.smithi007.stdout:test api_io_pp on pid 92653 2023-11-25T05:05:35.540 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_io_pp on pid 92653' 2023-11-25T05:05:35.540 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=92653 2023-11-25T05:05:35.540 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.540 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.540 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2023-11-25T05:05:35.540 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_asio 2023-11-25T05:05:35.540 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_asio' 2023-11-25T05:05:35.540 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.540 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.541 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.541 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_asio 2023-11-25T05:05:35.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.542 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.542 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.542 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.542 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.542 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.543 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.543 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.543 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.543 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.543 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.543 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.543 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.543 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.543 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.543 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.543 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.543 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.543 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.544 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.544 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.544 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.544 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.544 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.544 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.544 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.544 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.544 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.544 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.544 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.544 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.544 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.545 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.545 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.545 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.545 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.545 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.545 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.545 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.545 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.545 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.545 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.545 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.545 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.545 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.546 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.546 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.546 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.546 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.546 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.546 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.546 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.546 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.546 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.546 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.546 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.546 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.547 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.547 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.547 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_asio 2023-11-25T05:05:35.547 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92668 2023-11-25T05:05:35.547 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_asio on pid 92668' 2023-11-25T05:05:35.547 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2023-11-25T05:05:35.547 INFO:tasks.workunit.client.0.smithi007.stdout:test api_asio on pid 92668 2023-11-25T05:05:35.554 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=92668 2023-11-25T05:05:35.554 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.554 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.554 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_list 2023-11-25T05:05:35.554 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.554 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.554 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.554 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.554 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.555 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.555 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.555 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_list' 2023-11-25T05:05:35.555 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.555 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.555 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.555 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.555 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.555 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.555 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.555 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.555 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.555 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.556 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_list 2023-11-25T05:05:35.556 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.556 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.556 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.556 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.556 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.556 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.556 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.556 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.556 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.556 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.556 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.557 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.557 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.557 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.557 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.557 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.557 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.557 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.557 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.557 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.557 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.557 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.557 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.557 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.558 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.558 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.558 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.559 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.559 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.559 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.559 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.559 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.559 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.559 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.559 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.559 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.559 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.559 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.559 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.559 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.560 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.560 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.560 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/92624/exe 2023-11-25T05:05:35.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.560 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.561 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.561 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.561 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.562 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.562 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.562 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_list 2023-11-25T05:05:35.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.563 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.563 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.563 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.563 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.563 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.563 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.563 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.563 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.563 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92685 2023-11-25T05:05:35.563 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.563 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.563 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.564 INFO:tasks.workunit.client.0.smithi007.stdout:test api_list on pid 92685 2023-11-25T05:05:35.570 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_list on pid 92685' 2023-11-25T05:05:35.571 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=92685 2023-11-25T05:05:35.571 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.571 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.571 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.571 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.571 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.571 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.571 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.571 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.571 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.572 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.572 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.572 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.572 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.572 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.572 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_lock 2023-11-25T05:05:35.572 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.572 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.572 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.572 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.572 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.572 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.572 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_lock' 2023-11-25T05:05:35.573 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.573 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.573 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.573 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.573 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2023-11-25T05:05:35.573 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.573 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.573 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_aio 2023-11-25T05:05:35.573 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_aio.log 2023-11-25T05:05:35.573 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.573 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.573 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.573 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_aio: /' 2023-11-25T05:05:35.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.574 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.574 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.574 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.574 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.575 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.575 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.575 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.575 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.575 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.575 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.575 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.575 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.575 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_lock 2023-11-25T05:05:35.575 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.575 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.575 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.575 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.576 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.576 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.576 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.576 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.576 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.577 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.577 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.577 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.578 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.578 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.578 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.579 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.579 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.579 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/92631/exe 2023-11-25T05:05:35.579 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.579 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.579 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.579 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.579 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.579 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.579 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.579 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.579 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.579 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.580 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.580 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.580 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.580 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.580 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.580 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.580 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.580 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.580 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.580 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.580 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.580 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.580 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.581 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.581 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.581 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.581 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.581 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.581 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.581 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.581 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.581 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_lock 2023-11-25T05:05:35.581 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92708 2023-11-25T05:05:35.581 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.581 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.582 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.582 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_lock on pid 92708' 2023-11-25T05:05:35.582 INFO:tasks.workunit.client.0.smithi007.stdout:test api_lock on pid 92708 2023-11-25T05:05:35.591 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=92708 2023-11-25T05:05:35.591 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.592 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.592 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.592 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.592 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.592 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_aio_pp 2023-11-25T05:05:35.592 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.592 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.592 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_aio_pp: /' 2023-11-25T05:05:35.592 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.592 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.592 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.593 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.593 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.593 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.593 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.593 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2023-11-25T05:05:35.593 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.593 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.593 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.593 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.593 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.593 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.593 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.594 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.594 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.594 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.594 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.594 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.594 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.594 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.594 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.594 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.594 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.594 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.595 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.595 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.595 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.595 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.595 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.595 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.595 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.595 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.595 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.595 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.595 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.595 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.595 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.596 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.596 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.596 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.596 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.596 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.596 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.596 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.596 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.596 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.596 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.596 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.596 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.596 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_lock_pp 2023-11-25T05:05:35.597 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_lock_pp' 2023-11-25T05:05:35.597 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_aio_pp.log 2023-11-25T05:05:35.597 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/92640/exe 2023-11-25T05:05:35.597 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.597 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.597 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.597 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.597 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.597 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.597 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.597 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.597 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.597 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.598 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.598 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.598 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.598 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.598 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.598 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.598 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.598 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.598 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.598 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.598 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.598 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.599 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.599 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.599 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.599 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.599 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.599 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.599 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.599 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.599 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.599 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.599 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.599 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.600 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.600 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.600 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.600 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.600 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.600 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.600 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.600 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.600 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.600 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.600 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.600 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.600 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.601 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.601 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.601 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.601 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.601 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.601 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.601 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.601 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.601 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.601 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.601 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.601 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.601 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.602 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.602 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.602 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.602 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.602 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.602 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.602 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.602 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.602 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.602 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.602 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.602 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.603 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.603 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_lock_pp 2023-11-25T05:05:35.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.603 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.603 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.604 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.604 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.604 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_io.log 2023-11-25T05:05:35.604 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_io 2023-11-25T05:05:35.604 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_io: /' 2023-11-25T05:05:35.604 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.604 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.604 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.604 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.604 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/92653/exe 2023-11-25T05:05:35.604 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.604 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_lock_pp 2023-11-25T05:05:35.604 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92736 2023-11-25T05:05:35.605 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_lock_pp on pid 92736' 2023-11-25T05:05:35.605 INFO:tasks.workunit.client.0.smithi007.stdout:test api_lock_pp on pid 92736 2023-11-25T05:05:35.612 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=92736 2023-11-25T05:05:35.612 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.612 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.613 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.613 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.613 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.613 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2023-11-25T05:05:35.613 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.613 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.613 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.613 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.613 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_misc 2023-11-25T05:05:35.613 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.613 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.613 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.614 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.614 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_misc' 2023-11-25T05:05:35.614 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.614 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.614 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.614 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.614 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.614 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.614 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.614 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.614 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.614 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.615 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.615 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.615 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.615 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.615 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.615 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.615 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.615 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.615 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.615 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.615 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.615 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.615 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.616 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.616 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.616 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.616 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.616 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.616 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.616 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.616 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.616 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.616 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.616 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.616 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.616 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.617 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.617 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.617 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.617 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.617 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.617 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.617 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.617 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.617 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.617 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.617 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.617 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.617 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.618 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.618 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.618 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.618 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.618 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.618 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.618 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.618 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.618 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.618 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.618 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.618 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.619 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.619 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.619 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.619 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.619 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.619 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.619 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.619 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.619 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.619 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.619 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.619 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.619 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.620 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.620 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.620 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.620 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.620 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.620 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.620 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.620 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_io_pp: /' 2023-11-25T05:05:35.620 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.620 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.620 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.620 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.620 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.620 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.621 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.621 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.621 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.621 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_io_pp 2023-11-25T05:05:35.621 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.621 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_misc 2023-11-25T05:05:35.621 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/92668/exe 2023-11-25T05:05:35.621 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.621 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_io_pp.log 2023-11-25T05:05:35.621 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_misc 2023-11-25T05:05:35.621 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92754 2023-11-25T05:05:35.621 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_misc on pid 92754' 2023-11-25T05:05:35.622 INFO:tasks.workunit.client.0.smithi007.stdout:test api_misc on pid 92754 2023-11-25T05:05:35.627 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=92754 2023-11-25T05:05:35.628 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.628 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.628 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.628 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2023-11-25T05:05:35.628 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.628 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.628 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.628 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.628 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.628 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.628 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.628 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.629 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.629 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.629 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.629 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.629 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.629 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.629 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.629 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.629 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.629 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.629 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.629 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.629 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.630 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.630 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.631 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.631 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.631 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.631 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.631 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.631 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.631 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.631 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_misc_pp 2023-11-25T05:05:35.631 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_misc_pp' 2023-11-25T05:05:35.631 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_asio.log 2023-11-25T05:05:35.631 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.631 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.631 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.632 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.632 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.632 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.632 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_misc_pp 2023-11-25T05:05:35.632 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.632 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.632 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.632 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.632 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.632 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.632 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.632 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.632 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.633 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.633 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.633 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.633 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.633 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.633 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.633 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.633 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.633 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.633 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.633 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.633 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/92685/exe 2023-11-25T05:05:35.633 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_misc_pp 2023-11-25T05:05:35.634 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92768 2023-11-25T05:05:35.634 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_misc_pp on pid 92768' 2023-11-25T05:05:35.634 INFO:tasks.workunit.client.0.smithi007.stdout:test api_misc_pp on pid 92768 2023-11-25T05:05:35.642 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.642 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=92768 2023-11-25T05:05:35.642 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.642 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.642 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2023-11-25T05:05:35.642 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.643 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.643 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.643 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.643 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.643 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.643 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.643 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.643 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.643 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.643 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.643 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_tier_pp 2023-11-25T05:05:35.643 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.644 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.644 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.644 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.644 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.644 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_tier_pp' 2023-11-25T05:05:35.644 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.644 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.644 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.644 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.644 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.644 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.644 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.644 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.645 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.645 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.645 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.645 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.645 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.645 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.645 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.645 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.645 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.645 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.645 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.645 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.645 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.646 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.646 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.646 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.646 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.646 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.646 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.646 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.646 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.646 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.646 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_asio 2023-11-25T05:05:35.646 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.646 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.647 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.647 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.647 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.647 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.648 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.648 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.648 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.648 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.648 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.648 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.648 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.648 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.648 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.648 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.648 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.648 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_tier_pp 2023-11-25T05:05:35.649 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.649 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.649 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.649 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.649 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.649 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.649 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.649 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.649 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.649 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.649 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.649 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.650 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.650 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.650 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.650 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.650 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.650 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.650 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.650 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.650 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.650 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.650 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.650 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.650 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.651 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.651 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.651 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.651 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.651 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.651 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.651 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.651 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.651 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.651 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.651 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.651 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.651 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.652 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.652 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.652 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.652 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_tier_pp 2023-11-25T05:05:35.652 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92785 2023-11-25T05:05:35.652 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_tier_pp on pid 92785' 2023-11-25T05:05:35.652 INFO:tasks.workunit.client.0.smithi007.stdout:test api_tier_pp on pid 92785 2023-11-25T05:05:35.661 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=92785 2023-11-25T05:05:35.661 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.661 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.661 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.662 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2023-11-25T05:05:35.662 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.662 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.662 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.662 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.662 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.662 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.662 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.662 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.662 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_asio: /' 2023-11-25T05:05:35.662 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.662 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.663 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.663 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.663 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.663 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.663 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.663 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.663 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.663 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.663 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.663 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.663 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.663 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_pool 2023-11-25T05:05:35.663 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.664 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.664 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_pool' 2023-11-25T05:05:35.664 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.664 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.664 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_list 2023-11-25T05:05:35.664 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.665 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_list.log 2023-11-25T05:05:35.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.665 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.665 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.665 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.665 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.665 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.666 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.666 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.666 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.666 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.666 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.666 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.666 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.666 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.666 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.666 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.666 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.666 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.667 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.667 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.667 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.667 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.667 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.667 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.667 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.667 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.667 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.667 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.667 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.667 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.667 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.668 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.668 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.668 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.668 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.668 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.668 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.668 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.668 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.668 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.668 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.668 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.668 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.668 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.669 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.669 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.669 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.669 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.669 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.669 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.669 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.669 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.669 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.669 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_pool 2023-11-25T05:05:35.669 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.669 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.669 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.670 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.670 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_list: /' 2023-11-25T05:05:35.670 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.670 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.670 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.670 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.670 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.670 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_pool 2023-11-25T05:05:35.670 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92811 2023-11-25T05:05:35.670 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_pool on pid 92811' 2023-11-25T05:05:35.671 INFO:tasks.workunit.client.0.smithi007.stdout:test api_pool on pid 92811 2023-11-25T05:05:35.683 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=92811 2023-11-25T05:05:35.683 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.683 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.683 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.683 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2023-11-25T05:05:35.683 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.683 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.683 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.683 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.684 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.684 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.684 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.684 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.684 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.684 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.684 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.684 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.684 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.684 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.684 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.685 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.685 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.685 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.685 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.685 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.685 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.685 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.685 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.685 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.685 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.685 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.685 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.685 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.686 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.686 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.686 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.686 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.686 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.686 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.686 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.686 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.686 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.686 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.686 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_snapshots 2023-11-25T05:05:35.686 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.687 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.687 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_snapshots' 2023-11-25T05:05:35.687 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.687 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.687 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.687 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.687 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.687 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.687 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.687 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.687 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.687 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.687 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.688 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.688 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.688 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.688 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.688 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.688 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.688 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.688 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/92754/exe 2023-11-25T05:05:35.688 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.688 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.688 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.688 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.688 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.689 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.689 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.689 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.689 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.689 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.689 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.689 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.689 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.689 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.689 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.689 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.689 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.690 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.690 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.690 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.690 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.690 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.690 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.691 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/92708/exe 2023-11-25T05:05:35.691 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_snapshots 2023-11-25T05:05:35.691 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.691 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.691 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.691 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.691 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.691 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.691 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.691 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.692 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.692 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.692 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.692 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.692 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.692 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.692 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.692 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.692 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.692 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.692 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.692 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.692 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_lock.log 2023-11-25T05:05:35.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.693 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.693 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_misc 2023-11-25T05:05:35.693 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.693 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.693 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.693 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.694 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.695 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.695 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.695 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.695 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.695 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.695 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.695 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.695 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.695 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_misc.log 2023-11-25T05:05:35.695 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_snapshots 2023-11-25T05:05:35.695 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.695 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.695 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92854 2023-11-25T05:05:35.696 INFO:tasks.workunit.client.0.smithi007.stdout:test api_snapshots on pid 92854 2023-11-25T05:05:35.708 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_snapshots on pid 92854' 2023-11-25T05:05:35.708 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=92854 2023-11-25T05:05:35.708 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.708 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.708 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.708 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.708 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.708 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.709 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_misc: /' 2023-11-25T05:05:35.709 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.709 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.709 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.709 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.709 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.709 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.709 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.709 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.709 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.709 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.709 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.710 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.710 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_lock: /' 2023-11-25T05:05:35.710 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_lock 2023-11-25T05:05:35.710 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.710 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.710 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.711 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.711 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.711 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.711 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.711 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.711 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.711 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.711 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.711 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.711 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.711 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.711 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.711 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.712 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.712 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.712 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.712 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.712 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.712 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.712 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.712 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.712 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.712 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.712 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.712 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.712 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.713 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.713 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.713 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.713 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.713 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.713 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2023-11-25T05:05:35.713 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.713 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.713 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.713 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.713 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.713 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.714 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_snapshots_pp 2023-11-25T05:05:35.714 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_snapshots_pp' 2023-11-25T05:05:35.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_snapshots_pp 2023-11-25T05:05:35.714 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.714 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.714 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.714 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.714 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.715 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.715 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.715 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.715 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.715 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.715 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.715 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.715 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.715 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.715 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.715 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.715 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.715 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.716 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.716 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.716 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.716 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.716 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.716 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.716 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.716 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.716 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.716 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.716 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.716 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.716 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.717 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/92768/exe 2023-11-25T05:05:35.717 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.717 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/92736/exe 2023-11-25T05:05:35.717 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.717 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.717 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.717 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.717 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.717 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.717 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.717 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.717 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.718 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.718 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.718 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.718 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.718 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.718 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.718 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.718 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.718 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.718 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.718 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_misc_pp.log 2023-11-25T05:05:35.718 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.718 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.719 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.719 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.719 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.719 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.719 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.719 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.719 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.719 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.719 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.719 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.719 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.719 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.720 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.720 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.720 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_misc_pp: /' 2023-11-25T05:05:35.720 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.720 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.720 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.720 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.720 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.720 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.720 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.720 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.720 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.720 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.721 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.721 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.721 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.721 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.721 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.721 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_misc_pp 2023-11-25T05:05:35.721 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.721 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.721 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.721 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.721 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.721 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.721 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.722 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.722 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.722 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.722 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.722 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_snapshots_pp 2023-11-25T05:05:35.722 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.722 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92902 2023-11-25T05:05:35.722 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_snapshots_pp on pid 92902' 2023-11-25T05:05:35.722 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.722 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.722 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.722 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.722 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.723 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.723 INFO:tasks.workunit.client.0.smithi007.stdout:test api_snapshots_pp on pid 92902 2023-11-25T05:05:35.733 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.733 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=92902 2023-11-25T05:05:35.733 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.733 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.733 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_lock_pp 2023-11-25T05:05:35.733 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/92785/exe 2023-11-25T05:05:35.733 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.733 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.733 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_lock_pp.log 2023-11-25T05:05:35.734 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.734 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_lock_pp: /' 2023-11-25T05:05:35.734 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.735 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.735 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.735 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.735 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.735 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.735 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.735 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.735 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.735 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.735 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.735 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.735 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.735 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.735 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.736 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.736 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.736 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_tier_pp 2023-11-25T05:05:35.736 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.736 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.736 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.736 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.736 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_tier_pp.log 2023-11-25T05:05:35.736 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2023-11-25T05:05:35.736 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_tier_pp: /' 2023-11-25T05:05:35.736 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.736 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.736 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.737 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.737 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.737 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.737 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.737 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.737 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.737 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.737 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.737 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.737 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.737 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.737 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.737 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.737 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.738 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.738 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.738 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.738 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.738 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.738 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.738 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.738 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.738 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.738 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.738 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.738 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_stat 2023-11-25T05:05:35.738 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/92811/exe 2023-11-25T05:05:35.738 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_stat' 2023-11-25T05:05:35.739 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.739 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.740 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.740 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.740 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.740 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.740 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.740 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.740 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.740 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.740 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.740 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.740 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.740 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.740 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.740 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.741 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.741 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.741 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.741 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.741 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.741 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.742 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.742 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.742 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.742 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.742 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.742 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.742 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.742 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.742 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.742 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.743 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2023-11-25T05:05:35.743 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [----------] Global test environment set-up. 2023-11-25T05:05:35.743 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [----------] 12 tests from AsioRados 2023-11-25T05:05:35.743 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.618+0000 7faec4958780 10 monclient: get_monmap_and_config 2023-11-25T05:05:35.743 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.618+0000 7faec4958780 10 monclient: build_initial_monmap 2023-11-25T05:05:35.743 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.618+0000 7faec4958780 10 monclient: monmap: 2023-11-25T05:05:35.743 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 0 2023-11-25T05:05:35.743 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:05:35.743 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 0.000000 2023-11-25T05:05:35.743 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 0.000000 2023-11-25T05:05:35.743 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 0 (unknown) 2023-11-25T05:05:35.744 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2023-11-25T05:05:35.744 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:05:35.744 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:05:35.744 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:05:35.744 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.744 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.618+0000 7faec4958780 1 Processor -- start 2023-11-25T05:05:35.744 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.618+0000 7faec4958780 1 -- start start 2023-11-25T05:05:35.744 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.618+0000 7faec4958780 10 monclient: init 2023-11-25T05:05:35.744 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 10 monclient: _reopen_session rank -1 2023-11-25T05:05:35.745 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:05:35.745 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55f7e29b78a0 0x55f7e2a48ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:05:35.745 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 10 monclient(hunting): picked mon.noname-c con 0x55f7e29b78a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:05:35.745 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7e29bb5f0 0x55f7e29bd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:05:35.745 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 10 monclient(hunting): picked mon.noname-b con 0x55f7e29bb5f0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:05:35.745 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7e2a4b7e0 0x55f7e2ae5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:05:35.745 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 10 monclient(hunting): picked mon.noname-a con 0x55f7e2a4b7e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:05:35.745 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 10 monclient(hunting): start opening mon connection 2023-11-25T05:05:35.745 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7e28ce970 con 0x55f7e2a4b7e0 2023-11-25T05:05:35.745 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 10 monclient(hunting): start opening mon connection 2023-11-25T05:05:35.746 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7e28ce610 con 0x55f7e29bb5f0 2023-11-25T05:05:35.746 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 10 monclient(hunting): start opening mon connection 2023-11-25T05:05:35.746 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f7e28ae4e0 con 0x55f7e29b78a0 2023-11-25T05:05:35.746 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 10 monclient(hunting): _renew_subs 2023-11-25T05:05:35.746 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.619+0000 7faec4958780 10 monclient(hunting): authenticate will time out at 1461.862624s 2023-11-25T05:05:35.746 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_pool: /' 2023-11-25T05:05:35.747 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.623+0000 7faec1c5b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7e29bb5f0 0x55f7e29bd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:05:35.747 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.624+0000 7faec2c5d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7e2a4b7e0 0x55f7e2ae5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:05:35.747 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.624+0000 7faec1c5b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7e29bb5f0 0x55f7e29bd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:56216/0 (socket says 172.21.15.7:56216) 2023-11-25T05:05:35.747 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.625+0000 7faec1c5b640 1 -- 172.21.15.7:0/1870328067 learned_addr learned my addr 172.21.15.7:0/1870328067 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:05:35.747 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.625+0000 7faec1c5b640 10 monclient(hunting): get_auth_request con 0x55f7e29bb5f0 auth_method 0 2023-11-25T05:05:35.747 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.625+0000 7faec1c5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:05:35.747 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.625+0000 7faec1c5b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:05:35.747 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.625+0000 7faec1c5b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:05:35.747 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.625+0000 7faec2c5d640 10 monclient(hunting): get_auth_request con 0x55f7e2a4b7e0 auth_method 0 2023-11-25T05:05:35.747 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.625+0000 7faec2c5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:05:35.747 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.625+0000 7faec2c5d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:05:35.747 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.625+0000 7faec2c5d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:05:35.748 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.625+0000 7faec1c5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:05:35.748 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.625+0000 7faec1c5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:05:35.748 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.625+0000 7faec1c5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:05:35.748 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.626+0000 7faec2c5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:05:35.748 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.626+0000 7faec2c5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:05:35.748 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.626+0000 7faec2c5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:05:35.748 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.626+0000 7faec1c5b640 10 monclient(hunting): handle_auth_done global_id 4429 payload 274 2023-11-25T05:05:35.748 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.626+0000 7faec1c5b640 1 -- 172.21.15.7:0/1870328067 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55f7e29b78a0 msgr2=0x55f7e2a48ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:05:35.748 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.626+0000 7faec1c5b640 1 --2- 172.21.15.7:0/1870328067 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55f7e29b78a0 0x55f7e2a48ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:05:35.748 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.626+0000 7faec1c5b640 1 -- 172.21.15.7:0/1870328067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7e2a4b7e0 msgr2=0x55f7e2ae5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:05:35.753 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_pool.log 2023-11-25T05:05:35.753 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.753 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_pool 2023-11-25T05:05:35.754 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.754 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.754 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.754 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.754 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.754 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.754 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.754 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.754 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.754 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.754 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.754 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.755 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.755 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.755 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.755 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.756 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_stat 2023-11-25T05:05:35.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.756 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.756 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.756 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.757 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.757 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.757 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.757 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.757 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.757 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.758 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.758 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_stat 2023-11-25T05:05:35.758 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.758 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.758 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.758 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.758 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.758 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92976 2023-11-25T05:05:35.758 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_stat on pid 92976' 2023-11-25T05:05:35.759 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.627+0000 7faec1c5b640 1 --2- 172.21.15.7:0/1870328067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7e2a4b7e0 0x55f7e2ae5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0test api_stat on pid 92976 2023-11-25T05:05:35.763 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=92976 2023-11-25T05:05:35.763 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.763 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.763 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2023-11-25T05:05:35.763 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_stat_pp 2023-11-25T05:05:35.763 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_stat_pp' 2023-11-25T05:05:35.763 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.763 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.763 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.763 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.763 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.764 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.764 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.764 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.764 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.764 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.764 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.764 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.764 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.764 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.764 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.764 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.764 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.764 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.764 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.765 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.765 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.765 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.765 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.765 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.765 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.765 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.765 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.765 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.765 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.765 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.765 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.765 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.766 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.766 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.766 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.766 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.766 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.766 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.766 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.766 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.766 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.766 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.766 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.767 INFO:tasks.workunit.client.0.smithi007.stdout:).stop 2023-11-25T05:05:35.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.627+0000 7faec1c5b640 10 monclient: _finish_hunting 0 2023-11-25T05:05:35.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.627+0000 7faec1c5b640 1 monclient: found mon.noname-b 2023-11-25T05:05:35.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.627+0000 7faec1c5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:05:35.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.627+0000 7faec1c5b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:05:35.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.627+0000 7faec1c5b640 1 -- 172.21.15.7:0/1870328067 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc00b2d0 con 0x55f7e29bb5f0 2023-11-25T05:05:35.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.635+0000 7faec1c5b640 1 --2- 172.21.15.7:0/1870328067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7e29bb5f0 0x55f7e29bd2f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7faebc000f80 tx=0x7faebc04d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb255a6836136f75 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:05:35.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec0c59640 1 -- 172.21.15.7:0/1870328067 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc053b00 con 0x55f7e29bb5f0 2023-11-25T05:05:35.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec0c59640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:05:35.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec0c59640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:05:35.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec0c59640 10 monclient: dump: 2023-11-25T05:05:35.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 1 2023-11-25T05:05:35.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:05:35.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:05:35.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:05:35.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 18 (reef) 2023-11-25T05:05:35.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2023-11-25T05:05:35.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:05:35.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:05:35.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:05:35.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec0c59640 10 monclient: _finish_auth 0 2023-11-25T05:05:35.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec0c59640 10 monclient: _check_auth_tickets 2023-11-25T05:05:35.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec0c59640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:35.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec0c59640 1 -- 172.21.15.7:0/1870328067 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faebc00b050 con 0x55f7e29bb5f0 2023-11-25T05:05:35.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec0c59640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:05:35.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec0c59640 1 -- 172.21.15.7:0/1870328067 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc053e60 con 0x55f7e29bb5f0 2023-11-25T05:05:35.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec0c59640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:05:35.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec0c59640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:05:35.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec0c59640 10 monclient: dump: 2023-11-25T05:05:35.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 1 2023-11-25T05:05:35.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:05:35.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:05:35.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:05:35.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 18 (reef) 2023-11-25T05:05:35.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2023-11-25T05:05:35.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:05:35.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:05:35.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:05:35.773 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.773 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.773 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.774 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.774 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.774 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.774 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.774 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.774 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.774 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.774 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.774 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.774 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.774 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.774 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.775 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.775 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_stat_pp 2023-11-25T05:05:35.775 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.776 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.776 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.776 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.776 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.776 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.776 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_stat_pp 2023-11-25T05:05:35.776 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93006 2023-11-25T05:05:35.776 INFO:tasks.workunit.client.0.smithi007.stdout: test api_stat_pp on pid 93006 2023-11-25T05:05:35.777 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_stat_pp on pid 93006' 2023-11-25T05:05:35.777 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=93006 2023-11-25T05:05:35.777 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.777 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.778 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.778 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec4958780 5 monclient: authenticate success, global_id 4429 2023-11-25T05:05:35.778 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec4958780 10 monclient: get_monmap_and_config success 2023-11-25T05:05:35.778 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.662+0000 7faec4958780 4 set_mon_vals no callback set 2023-11-25T05:05:35.778 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.663+0000 7faec4958780 10 monclient: shutdown 2023-11-25T05:05:35.778 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.663+0000 7faec4958780 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:05:35.778 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.663+0000 7faec4958780 1 -- 172.21.15.7:0/1870328067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7e29bb5f0 msgr2=0x55f7e29bd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:05:35.778 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.663+0000 7faec4958780 1 --2- 172.21.15.7:0/1870328067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7e29bb5f0 0x55f7e29bd2f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7faebc000f80 tx=0x7faebc04d000 comp rx=0 tx=0).stop 2023-11-25T05:05:35.778 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.666+0000 7faec4958780 1 -- 172.21.15.7:0/1870328067 shutdown_connections 2023-11-25T05:05:35.778 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.666+0000 7faec4958780 1 --2- 172.21.15.7:0/1870328067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7e2a4b7e0 0x55f7e2ae5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:05:35.778 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.666+0000 7faec4958780 1 --2- 172.21.15.7:0/1870328067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7e29bb5f0 0x55f7e29bd2f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:05:35.779 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.666+0000 7faec4958780 1 --2- 172.21.15.7:0/1870328067 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55f7e29b78a0 0x55f7e2a48ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:05:35.779 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.666+0000 7faec4958780 1 -- 172.21.15.7:0/1870328067 >> 172.21.15.7:0/1870328067 conn(0x55f7e29bebc0 msgr2=0x55f7e2a4acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:05:35.779 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.666+0000 7faec4958780 1 -- 172.21.15.7:0/1870328067 shutdown_connections 2023-11-25T05:05:35.779 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.666+0000 7faec4958780 1 -- 172.21.15.7:0/1870328067 wait complete. 2023-11-25T05:05:35.779 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.666+0000 7faec4958780 10 monclient: build_initial_monmap 2023-11-25T05:05:35.779 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.666+0000 7faec4958780 10 monclient: monmap: 2023-11-25T05:05:35.779 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 0 2023-11-25T05:05:35.779 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:05:35.779 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 0.000000 2023-11-25T05:05:35.779 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 0.000000 2023-11-25T05:05:35.779 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 0 (unknown) 2023-11-25T05:05:35.779 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2023-11-25T05:05:35.779 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:05:35.780 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:05:35.780 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:05:35.780 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.780 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.667+0000 7faec4958780 1 librados: starting msgr at 2023-11-25T05:05:35.780 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.667+0000 7faec4958780 1 librados: starting objecter 2023-11-25T05:05:35.780 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.667+0000 7faec4958780 1 Processor -- start 2023-11-25T05:05:35.780 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.667+0000 7faec4958780 1 -- start start 2023-11-25T05:05:35.780 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 1 librados: setting wanted keys 2023-11-25T05:05:35.780 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 1 librados: calling monclient init 2023-11-25T05:05:35.780 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 10 monclient: init 2023-11-25T05:05:35.780 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 10 monclient: _reopen_session rank -1 2023-11-25T05:05:35.782 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2023-11-25T05:05:35.782 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/92854/exe 2023-11-25T05:05:35.782 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.782 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.782 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.783 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_watch_notify 2023-11-25T05:05:35.783 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.783 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.783 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.783 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.783 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.783 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.783 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.783 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.783 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.783 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.783 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_watch_notify' 2023-11-25T05:05:35.783 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.784 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.784 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.784 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.784 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.784 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.784 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.784 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.784 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.784 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.784 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.785 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.785 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.785 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.785 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.785 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.785 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 10 monclient: _add_conns ranks=[1,0,2] 2023-11-25T05:05:35.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7e29b78a0 0x55f7e2a4a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:05:35.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 10 monclient(hunting): picked mon.noname-b con 0x55f7e29b78a0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:05:35.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7e29bb5f0 0x55f7e2a44a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:05:35.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 10 monclient(hunting): picked mon.noname-a con 0x55f7e29bb5f0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:05:35.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55f7e2a4b7e0 0x55f7e2a44fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:05:35.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 10 monclient(hunting): picked mon.noname-c con 0x55f7e2a4b7e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:05:35.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 10 monclient(hunting): start opening mon connection 2023-11-25T05:05:35.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7e2a45500 con 0x55f7e29bb5f0 2023-11-25T05:05:35.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 10 monclient(hunting): start opening mon connection 2023-11-25T05:05:35.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7e2ae5290 con 0x55f7e29b78a0 2023-11-25T05:05:35.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 10 monclient(hunting): start opening mon connection 2023-11-25T05:05:35.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f7e2ae5570 con 0x55f7e2a4b7e0 2023-11-25T05:05:35.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 10 monclient(hunting): _renew_subs 2023-11-25T05:05:35.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.668+0000 7faec4958780 10 monclient(hunting): authenticate will time out at 1461.911792s 2023-11-25T05:05:35.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.670+0000 7faec2c5d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55f7e2a4b7e0 0x55f7e2a44fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:05:35.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.671+0000 7faec2c5d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55f7e2a4b7e0 0x55f7e2a44fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:50716/0 (socket says 172.21.15.7:50716) 2023-11-25T05:05:35.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.671+0000 7faec2c5d640 1 -- 172.21.15.7:0/3026089490 learned_addr learned my addr 172.21.15.7:0/3026089490 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:05:35.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.671+0000 7faec2c5d640 10 monclient(hunting): get_auth_request con 0x55f7e2a4b7e0 auth_method 0 2023-11-25T05:05:35.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.671+0000 7faec2c5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:05:35.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.671+0000 7faec2c5d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:05:35.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.671+0000 7faec2c5d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:05:35.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.672+0000 7faec2c5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:05:35.795 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.795 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.795 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.795 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.796 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.796 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_snapshots 2023-11-25T05:05:35.796 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.796 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.797 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.797 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.797 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.797 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.797 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.797 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.797 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.797 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.797 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_watch_notify 2023-11-25T05:05:35.797 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.797 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.797 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.797 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.798 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.798 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.798 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.798 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.798 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_snapshots.log 2023-11-25T05:05:35.798 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.798 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.798 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.798 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.798 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_snapshots: /' 2023-11-25T05:05:35.798 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.798 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.799 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.799 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.799 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.799 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.799 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.799 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.799 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.799 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.799 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.799 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.799 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.799 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.799 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.800 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.800 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.800 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.800 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.800 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.800 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.800 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.800 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.800 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.800 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.800 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.800 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.800 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.800 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.801 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.801 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.801 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.801 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.801 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.801 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.801 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.801 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/92902/exe 2023-11-25T05:05:35.801 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.801 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.801 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.801 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.801 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.802 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.802 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.802 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.802 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.802 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.802 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.802 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.802 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.802 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.802 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.802 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.802 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.802 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.802 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.803 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.803 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.803 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.803 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.803 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.803 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.803 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_watch_notify 2023-11-25T05:05:35.803 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.803 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.803 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93060 2023-11-25T05:05:35.803 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_watch_notify on pid 93060' 2023-11-25T05:05:35.804 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.672+0000 7faec2c5d640 10 monclient(hunttest api_watch_notify on pid 93060 2023-11-25T05:05:35.810 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=93060 2023-11-25T05:05:35.810 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.810 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.810 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.810 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_snapshots_pp: /' 2023-11-25T05:05:35.810 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.810 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_snapshots_pp 2023-11-25T05:05:35.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_watch_notify_pp 2023-11-25T05:05:35.811 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_watch_notify_pp' 2023-11-25T05:05:35.811 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2023-11-25T05:05:35.811 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.811 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.811 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2023-11-25T05:05:35.811 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_watch_notify_pp 2023-11-25T05:05:35.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.811 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.812 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.812 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.812 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.812 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.812 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.812 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.812 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.812 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.812 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.812 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.812 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.812 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.813 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.813 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.813 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.813 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.813 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.813 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.813 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.813 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.813 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.813 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.813 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.813 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.813 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.814 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.814 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.814 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.814 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.814 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.814 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.814 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.814 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.814 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.814 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.814 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.814 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.815 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.815 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.815 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.815 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.815 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.815 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.815 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.815 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.815 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.815 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.815 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.815 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.815 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.816 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.816 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.816 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.816 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.816 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.816 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.816 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.816 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.816 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.816 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.816 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_watch_notify_pp 2023-11-25T05:05:35.816 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93082 2023-11-25T05:05:35.817 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_watch_notify_pp on pid 93082' 2023-11-25T05:05:35.817 INFO:tasks.workunit.client.0.smithi007.stdout:test api_watch_notify_pp on pid 93082 2023-11-25T05:05:35.823 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=93082 2023-11-25T05:05:35.824 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.824 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.824 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_cmd 2023-11-25T05:05:35.824 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.824 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.824 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.824 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_cmd' 2023-11-25T05:05:35.824 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.824 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.824 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.824 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.825 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.825 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.825 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.825 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2023-11-25T05:05:35.825 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.825 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.825 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.825 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.825 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.825 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.825 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.825 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.825 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_cmd 2023-11-25T05:05:35.826 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.826 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.826 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.826 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.826 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.826 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.826 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.826 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.826 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.826 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/92976/exe 2023-11-25T05:05:35.826 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.826 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.827 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.827 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.827 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.827 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.827 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.827 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.827 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.827 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.827 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.827 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.827 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.827 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.827 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.828 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.828 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.828 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.828 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.828 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.828 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.828 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.828 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.828 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.828 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.828 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.828 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.828 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.829 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.829 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.829 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.829 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.829 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.829 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.829 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.829 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.829 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.829 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.829 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.829 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.829 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.830 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.830 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.830 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.830 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.830 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.830 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.830 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.830 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_cmd 2023-11-25T05:05:35.830 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.830 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.830 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.830 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.831 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.831 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.831 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.831 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93094 2023-11-25T05:05:35.831 INFO:tasks.workunit.client.0.smithi007.stdout:test api_cmd on pid 93094 2023-11-25T05:05:35.838 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_cmd on pid 93094' 2023-11-25T05:05:35.838 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=93094 2023-11-25T05:05:35.838 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/93006/exe 2023-11-25T05:05:35.838 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.838 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.838 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.838 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.838 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.839 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.839 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.839 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.839 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.839 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.839 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.843 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.843 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.843 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.843 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.843 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.843 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.844 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.844 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.844 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.844 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.844 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.844 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.844 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.844 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.845 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.845 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_stat.log 2023-11-25T05:05:35.845 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2023-11-25T05:05:35.845 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_stat 2023-11-25T05:05:35.845 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.845 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.845 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.845 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.845 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.846 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.846 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.846 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.846 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.846 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.846 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.846 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.847 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_stat_pp 2023-11-25T05:05:35.847 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_stat: /' 2023-11-25T05:05:35.847 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.848 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.848 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.849 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.849 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.849 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.849 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.849 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.849 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.850 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_stat_pp: /' 2023-11-25T05:05:35.850 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_cmd_pp 2023-11-25T05:05:35.850 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_cmd_pp' 2023-11-25T05:05:35.850 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_stat_pp.log 2023-11-25T05:05:35.850 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.851 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.851 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.851 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.851 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.851 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.851 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.852 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.852 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.852 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.852 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.852 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.852 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.852 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.853 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.853 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.853 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.853 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.853 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.854 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.854 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.854 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.854 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.854 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.855 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.855 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.855 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.855 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.855 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.856 INFO:tasks.workunit.client.0.smithi007.stdout:ing): handle_auth_reply_more payload_len 9 2023-11-25T05:05:35.856 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.672+0000 7faec2c5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:05:35.857 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.681+0000 7faec245c640 1 --2- 172.21.15.7:0/3026089490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7e29b78a0 0x55f7e2a4a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:05:35.857 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.683+0000 7faec245c640 10 monclient(hunting): get_auth_request con 0x55f7e29b78a0 auth_method 0 2023-11-25T05:05:35.857 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.683+0000 7faec245c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:05:35.857 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.683+0000 7faec245c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:05:35.857 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.683+0000 7faec245c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:05:35.857 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.685+0000 7faec245c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:05:35.857 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.685+0000 7faec245c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:05:35.858 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.685+0000 7faec245c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:05:35.858 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.688+0000 7faec2c5d640 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2023-11-25T05:05:35.858 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.688+0000 7faec2c5d640 1 -- 172.21.15.7:0/3026089490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7e29b78a0 msgr2=0x55f7e2a4a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:05:35.858 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.688+0000 7faec2c5d640 1 --2- 172.21.15.7:0/3026089490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7e29b78a0 0x55f7e2a4a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:05:35.858 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.688+0000 7faec2c5d640 1 -- 172.21.15.7:0/3026089490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7e29bb5f0 msgr2=0x55f7e2a44a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:05:35.858 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.688+0000 7faec2c5d640 1 --2- 172.21.15.7:0/3026089490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7e29bb5f0 0x55f7e2a44a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:05:35.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.688+0000 7faec2c5d640 10 monclient: _finish_hunting 0 2023-11-25T05:05:35.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.688+0000 7faec2c5d640 1 monclient: found mon.noname-c 2023-11-25T05:05:35.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.688+0000 7faec2c5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:05:35.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.688+0000 7faec2c5d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:05:35.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.688+0000 7faec2c5d640 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7e28cd590 con 0x55f7e2a4b7e0 2023-11-25T05:05:35.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.697+0000 7faec2c5d640 1 --2- 172.21.15.7:0/3026089490 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55f7e2a4b7e0 0x55f7e2a44fc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faeb8009df0 tx=0x7faeb8009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:05:35.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.697+0000 7faeb37fe640 10 client.?.objecter ms_handle_connect 0x55f7e2a4b7e0 2023-11-25T05:05:35.860 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.697+0000 7faeb37fe640 10 client.?.objecter resend_mon_ops 2023-11-25T05:05:35.860 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb800f050 con 0x55f7e2a4b7e0 2023-11-25T05:05:35.860 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faeb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:05:35.860 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faeb37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:05:35.860 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faeb37fe640 10 monclient: dump: 2023-11-25T05:05:35.860 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 1 2023-11-25T05:05:35.860 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:05:35.861 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:05:35.861 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:05:35.861 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 18 (reef) 2023-11-25T05:05:35.861 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2023-11-25T05:05:35.861 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:05:35.861 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:05:35.861 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:05:35.861 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.861 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faeb37fe640 10 monclient: _finish_auth 0 2023-11-25T05:05:35.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faeb37fe640 10 monclient: _check_auth_tickets 2023-11-25T05:05:35.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faeb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:35.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb8010040 con 0x55f7e2a4b7e0 2023-11-25T05:05:35.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faeb37fe640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:05:35.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb8015680 con 0x55f7e2a4b7e0 2023-11-25T05:05:35.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faeb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:05:35.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faeb37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:05:35.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faeb37fe640 10 monclient: dump: 2023-11-25T05:05:35.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 1 2023-11-25T05:05:35.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:05:35.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:05:35.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:05:35.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 18 (reef) 2023-11-25T05:05:35.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2023-11-25T05:05:35.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:05:35.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:05:35.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:05:35.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faec0c59640 4 set_mon_vals no callback set 2023-11-25T05:05:35.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faec4958780 5 monclient: authenticate success, global_id 4355 2023-11-25T05:05:35.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faec4958780 10 monclient: _renew_subs 2023-11-25T05:05:35.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faec4958780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:05:35.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f7e28f4760 con 0x55f7e2a4b7e0 2023-11-25T05:05:35.866 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faec4958780 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:05:35.866 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faec4958780 10 monclient: _renew_subs 2023-11-25T05:05:35.866 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faec4958780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:05:35.866 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f7e28da770 con 0x55f7e2a4b7e0 2023-11-25T05:05:35.869 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_cmd_pp 2023-11-25T05:05:35.869 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.869 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.869 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.869 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.869 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.869 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.870 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.870 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.870 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.870 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.870 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_cmd_pp 2023-11-25T05:05:35.870 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93126 2023-11-25T05:05:35.871 INFO:tasks.workunit.client.0.smithi007.stdout: test api_cmd_pp on pid 93126 2023-11-25T05:05:35.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_cmd_pp on pid 93126' 2023-11-25T05:05:35.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=93126 2023-11-25T05:05:35.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2023-11-25T05:05:35.891 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_service 2023-11-25T05:05:35.891 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.891 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.891 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.892 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.893 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_service' 2023-11-25T05:05:35.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.893 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.893 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.893 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.893 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.893 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.893 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.894 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.894 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.894 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.894 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.894 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.894 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.894 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.894 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.894 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.894 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.894 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.894 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.894 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.895 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.895 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.895 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.895 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.895 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.895 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.895 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.895 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.895 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.895 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.895 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.895 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.896 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.896 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.896 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.896 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.896 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.896 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.896 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.896 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.896 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.896 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.896 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.896 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.896 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.897 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.897 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.897 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.897 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.897 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.897 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.897 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.897 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.897 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.897 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.897 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.897 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.897 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.898 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.898 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.898 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.898 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.898 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.898 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.898 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.898 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.898 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.898 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.898 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.898 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.899 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.899 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.899 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.899 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.899 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.899 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.899 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.899 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.899 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.899 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.899 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.899 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.899 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.900 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.900 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.900 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.900 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.900 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_service 2023-11-25T05:05:35.900 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.900 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.900 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.900 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.900 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.900 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.900 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.900 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.901 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.901 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.901 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.901 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.901 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.901 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.901 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.901 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.901 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.901 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.901 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.901 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.902 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.902 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.902 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.902 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.902 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.902 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.902 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.902 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.902 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.902 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.903 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.903 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.903 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.903 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.903 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.903 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.903 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.903 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.903 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_service 2023-11-25T05:05:35.903 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/93060/exe 2023-11-25T05:05:35.903 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93157 2023-11-25T05:05:35.903 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_service on pid 93157' 2023-11-25T05:05:35.904 INFO:tasks.workunit.client.0.smithi007.stdout:test api_service on pid 93157 2023-11-25T05:05:35.916 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=93157 2023-11-25T05:05:35.916 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.917 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_service_pp 2023-11-25T05:05:35.917 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_service_pp' 2023-11-25T05:05:35.917 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.917 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.917 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.917 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.917 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.918 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.918 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.918 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.918 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.918 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.918 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.918 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.918 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.918 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.918 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.918 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.918 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.919 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.919 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.919 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.919 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.919 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.919 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.919 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.919 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.919 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.919 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2023-11-25T05:05:35.919 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.919 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_watch_notify 2023-11-25T05:05:35.919 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.920 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.920 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.920 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.920 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.920 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.920 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.920 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.920 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.920 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.920 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.920 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.920 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.920 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.921 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.921 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.921 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.921 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.921 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.921 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.921 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.921 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.921 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.921 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.921 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.921 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.922 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.922 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.923 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.923 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.923 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/93094/exe 2023-11-25T05:05:35.924 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/93082/exe 2023-11-25T05:05:35.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.924 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.925 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.925 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.925 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.925 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.925 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.925 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.925 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.925 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.926 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.926 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_cmd.log 2023-11-25T05:05:35.926 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.926 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.926 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.926 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.926 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.926 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.926 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.926 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.926 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.926 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.926 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.927 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.927 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.927 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.927 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.927 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.927 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.927 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.927 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.927 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.927 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.927 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.927 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_cmd 2023-11-25T05:05:35.928 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_watch_notify_pp 2023-11-25T05:05:35.928 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2023-11-25T05:05:35.928 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.928 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.928 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.928 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.928 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.928 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.928 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.928 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.928 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.928 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.928 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.928 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.929 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.929 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.929 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.929 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.929 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.929 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.929 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.929 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_service_pp 2023-11-25T05:05:35.929 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_cmd: /' 2023-11-25T05:05:35.929 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_watch_notify.log 2023-11-25T05:05:35.929 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_service_pp 2023-11-25T05:05:35.929 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93192 2023-11-25T05:05:35.929 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_watch_notify: /' 2023-11-25T05:05:35.929 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_service_pp on pid 93192' 2023-11-25T05:05:35.930 INFO:tasks.workunit.client.0.smithi007.stdout:test api_service_pp on pid 93192 2023-11-25T05:05:35.934 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=93192 2023-11-25T05:05:35.935 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.935 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.935 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2023-11-25T05:05:35.935 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.935 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.935 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.936 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.936 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.936 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.937 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.937 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2023-11-25T05:05:35.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_c_write_operations 2023-11-25T05:05:35.937 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_c_write_operations' 2023-11-25T05:05:35.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_c_write_operations 2023-11-25T05:05:35.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.937 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.937 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.937 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.938 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.938 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.938 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.938 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.938 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.938 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.938 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.938 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.938 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.938 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.938 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.938 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.939 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.939 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.939 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.939 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.939 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.939 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.939 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.939 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.939 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.939 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.939 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.939 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.939 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.940 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.940 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.940 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.940 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.940 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.940 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.940 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.940 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.940 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.940 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.940 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_c_write_operations 2023-11-25T05:05:35.940 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93222 2023-11-25T05:05:35.940 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_c_write_operations on pid 93222' 2023-11-25T05:05:35.941 INFO:tasks.workunit.client.0.smithi007.stdout:test api_c_write_operations on pid 93222 2023-11-25T05:05:35.946 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=93222 2023-11-25T05:05:35.946 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.946 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.946 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.946 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.946 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2023-11-25T05:05:35.947 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.947 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_c_read_operations 2023-11-25T05:05:35.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.947 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.947 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.948 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_c_read_operations' 2023-11-25T05:05:35.948 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.948 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.948 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.948 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.948 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.948 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.948 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.948 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.948 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.948 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.948 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_c_read_operations 2023-11-25T05:05:35.948 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.948 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.949 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.949 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.949 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.949 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.949 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.949 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.949 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.949 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.950 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.950 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.950 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.950 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.950 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.950 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.950 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.950 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.950 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.950 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.951 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.951 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.951 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.951 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.951 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.951 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.951 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.951 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.951 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.951 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.951 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.951 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.951 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.952 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.952 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.952 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.952 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.952 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.952 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.952 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.952 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.952 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.952 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.952 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.952 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_c_read_operations 2023-11-25T05:05:35.953 INFO:tasks.workunit.client.0.smithi007.stdout:test api_c_read_operations on pid 93238 2023-11-25T05:05:35.963 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93238 2023-11-25T05:05:35.963 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_c_read_operations on pid 93238' 2023-11-25T05:05:35.963 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=93238 2023-11-25T05:05:35.963 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.963 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.963 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2023-11-25T05:05:35.963 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.963 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.964 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.964 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.964 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.964 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.964 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.964 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.964 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.964 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.964 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.964 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.964 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.965 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.965 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.965 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.965 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.965 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.965 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.965 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.965 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.965 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.965 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.965 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.965 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.965 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.966 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.966 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.966 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.966 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.966 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.966 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.966 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.966 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.966 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.966 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/93126/exe 2023-11-25T05:05:35.966 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.966 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.966 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.967 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.967 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.967 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.967 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.967 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_cls_remote_reads 2023-11-25T05:05:35.967 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.967 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_cls_remote_reads' 2023-11-25T05:05:35.967 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.967 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.967 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.967 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.967 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.968 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.968 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.968 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.968 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.968 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.968 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.968 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.968 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.968 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.968 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.968 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.968 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.968 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.969 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.969 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.969 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.969 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.969 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.969 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.970 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.970 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.970 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.970 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.971 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.971 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_cls_remote_reads 2023-11-25T05:05:35.971 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_cmd_pp 2023-11-25T05:05:35.971 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.971 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.971 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.971 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.971 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.972 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.972 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.972 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.972 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.972 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.972 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.972 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.973 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.973 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.973 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2023-11-25T05:05:35.973 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_cmd_pp: /' 2023-11-25T05:05:35.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:35.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:35.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:35.973 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.973 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:35.973 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.973 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:35.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:35.974 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:35.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:35.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:35.974 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:35.974 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:35.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:35.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:35.974 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/93157/exe 2023-11-25T05:05:35.976 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.976 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.977 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.977 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.977 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.978 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:35.978 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:35.978 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:35.978 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.978 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:35.978 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.978 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:35.978 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:35.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:35.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:35.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:35.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:35.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:35.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:35.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:35.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:35.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.980 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.980 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_cls_remote_reads 2023-11-25T05:05:35.980 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93294 2023-11-25T05:05:35.980 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_cls_remote_reads on pid 93294' 2023-11-25T05:05:35.980 INFO:tasks.workunit.client.0.smithi007.stdout:test api_cls_remote_reads on pid 93294 2023-11-25T05:05:35.988 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=93294 2023-11-25T05:05:35.988 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:35.988 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:35.988 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.988 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/93192/exe 2023-11-25T05:05:35.988 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2023-11-25T05:05:35.988 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:35.989 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:35.989 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:35.989 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:35.989 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.989 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:35.989 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.989 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:35.989 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:35.989 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.989 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:35.989 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.989 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.989 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.990 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.990 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.990 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.990 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.990 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.990 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.990 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.990 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.990 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.990 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.990 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.990 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.991 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.991 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:35.991 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.991 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:35.991 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:35.991 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.991 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:35.991 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s list_parallel 2023-11-25T05:05:35.991 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:35.991 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' list_parallel' 2023-11-25T05:05:35.991 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:35.991 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:35.991 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:35.991 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:35.992 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:35.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:35.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:35.992 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:35.992 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:35.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:35.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:35.992 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:35.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:35.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:35.993 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:35.993 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:35.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:35.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:35.993 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.993 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:35.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:35.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:35.993 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:35.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo list_parallel 2023-11-25T05:05:35.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:35.994 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_service 2023-11-25T05:05:35.994 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:35.994 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:35.994 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_service_pp: /' 2023-11-25T05:05:35.994 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:35.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:35.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:35.994 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:35.994 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:35.995 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:35.995 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:35.995 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:35.995 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:35.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:35.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.996 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:35.996 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:35.996 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:35.996 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:35.996 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:35.996 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:35.996 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:35.996 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_service.log 2023-11-25T05:05:35.996 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_service_pp 2023-11-25T05:05:35.996 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_service: /' 2023-11-25T05:05:35.997 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_service_pp.log 2023-11-25T05:05:35.997 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=list_parallel 2023-11-25T05:05:35.997 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93333 2023-11-25T05:05:35.997 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test list_parallel on pid 93333' 2023-11-25T05:05:35.997 INFO:tasks.workunit.client.0.smithi007.stdout:test list_parallel on pid 93333 2023-11-25T05:05:36.002 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=93333 2023-11-25T05:05:36.002 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:36.002 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:36.002 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:36.002 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:36.002 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:36.003 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:36.003 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:36.003 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:36.003 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:36.003 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:36.003 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:36.003 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:36.003 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:36.003 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:36.003 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s open_pools_parallel 2023-11-25T05:05:36.003 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.003 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:36.003 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.004 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:36.004 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:36.004 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:36.004 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:36.004 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:36.004 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.004 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.004 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:36.004 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:36.004 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.004 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.004 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:36.004 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:36.005 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:36.005 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:36.005 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.005 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:36.005 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.005 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:36.005 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:36.005 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:36.005 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.005 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:36.005 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.006 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:36.006 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' open_pools_parallel' 2023-11-25T05:05:36.006 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:36.006 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:36.006 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2023-11-25T05:05:36.006 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo open_pools_parallel 2023-11-25T05:05:36.006 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:36.006 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:36.006 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:36.006 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:36.006 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:36.006 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.006 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:36.007 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/93222/exe 2023-11-25T05:05:36.007 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:36.007 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:36.007 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:36.007 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:36.007 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:36.007 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:36.007 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:36.007 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:36.007 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.007 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:36.007 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=open_pools_parallel 2023-11-25T05:05:36.007 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.008 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:36.008 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:36.008 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.008 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93345 2023-11-25T05:05:36.008 INFO:tasks.workunit.client.0.smithi007.stdout:test open_pools_parallel on pid 93345 2023-11-25T05:05:36.016 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test open_pools_parallel on pid 93345' 2023-11-25T05:05:36.016 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=93345 2023-11-25T05:05:36.016 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:36.016 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-25T05:05:36.016 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:36.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:36.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:36.017 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:36.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s delete_pools_parallel 2023-11-25T05:05:36.017 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:36.017 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:36.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:36.017 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' delete_pools_parallel' 2023-11-25T05:05:36.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:36.017 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:36.017 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2023-11-25T05:05:36.017 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:36.017 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:36.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:36.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:36.018 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:36.018 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:36.018 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:36.018 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:36.018 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:36.018 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:36.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:36.018 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:36.018 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:36.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:36.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:36.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:36.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:36.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:36.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:36.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:36.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:36.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:36.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:36.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:36.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:36.020 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_c_write_operations: /' 2023-11-25T05:05:36.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:36.020 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:36.020 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2023-11-25T05:05:36.020 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:36.020 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:36.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:36.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:36.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:36.020 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:36.020 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:36.021 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:36.021 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:36.021 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_c_write_operations 2023-11-25T05:05:36.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:36.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:36.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:36.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.021 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/93238/exe 2023-11-25T05:05:36.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:36.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:36.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:36.022 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:36.022 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:36.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:36.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:36.022 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:36.022 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:36.022 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:36.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:36.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.023 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:36.023 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:36.023 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:36.023 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo delete_pools_parallel 2023-11-25T05:05:36.023 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2023-11-25T05:05:36.023 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:36.023 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:36.023 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:36.023 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:36.023 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:36.023 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:36.023 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:36.023 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:36.023 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:36.024 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:36.024 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_c_read_operations: /' 2023-11-25T05:05:36.024 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_c_read_operations 2023-11-25T05:05:36.024 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=delete_pools_parallel 2023-11-25T05:05:36.024 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93372 2023-11-25T05:05:36.024 INFO:tasks.workunit.client.0.smithi007.stdout:test delete_pools_parallel on pid 93372 2023-11-25T05:05:36.052 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test delete_pools_parallel on pid 93372' 2023-11-25T05:05:36.052 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2023-11-25T05:05:36.052 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=93372 2023-11-25T05:05:36.052 INFO:tasks.workunit.client.0.smithi007.stderr:+ ret=0 2023-11-25T05:05:36.052 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2023-11-25T05:05:36.052 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:05:36.052 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92785 2023-11-25T05:05:36.052 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 92785 2023-11-25T05:05:36.052 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2023-11-25T05:05:36.053 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2023-11-25T05:05:36.053 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2023-11-25T05:05:36.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2023-11-25T05:05:36.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2023-11-25T05:05:36.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2023-11-25T05:05:36.053 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2023-11-25T05:05:36.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2023-11-25T05:05:36.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2023-11-25T05:05:36.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-25T05:05:36.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-25T05:05:36.054 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-25T05:05:36.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-25T05:05:36.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-25T05:05:36.054 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:36.054 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-25T05:05:36.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-25T05:05:36.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-25T05:05:36.054 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:36.054 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:36.055 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:36.055 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:36.055 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:36.055 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:36.055 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:36.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:36.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:36.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:36.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:36.055 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:36.056 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:36.056 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:36.056 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:36.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:36.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:36.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:36.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:36.056 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:36.056 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:36.056 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:36.057 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:36.057 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:36.057 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:36.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:36.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:36.057 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:36.057 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:36.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:36.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:36.058 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:36.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:36.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:36.058 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:36.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:36.058 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2023-11-25T05:05:36.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:36.058 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-25T05:05:36.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:36.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:36.059 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:36.059 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-25T05:05:36.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-25T05:05:36.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2023-11-25T05:05:36.059 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2023-11-25T05:05:36.059 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2023-11-25T05:05:36.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-25T05:05:36.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.060 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-25T05:05:36.060 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2023-11-25T05:05:36.060 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2023-11-25T05:05:36.060 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.060 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-25T05:05:36.060 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.060 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-25T05:05:36.060 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:36.060 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:36.060 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:36.060 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:36.060 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:36.060 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:36.060 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:36.061 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:36.061 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:36.061 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:36.061 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.061 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:36.061 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.061 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:36.061 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:36.061 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:36.061 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:36.061 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:36.061 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:36.061 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:36.061 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.062 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:36.062 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.062 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:36.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:36.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:36.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:36.062 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:36.062 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:36.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:36.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:36.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:36.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:36.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:36.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:36.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:36.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:36.063 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.063 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:36.063 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.063 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:36.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:36.063 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.063 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:36.063 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.063 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:36.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:36.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-25T05:05:36.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-25T05:05:36.064 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-25T05:05:36.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-25T05:05:36.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-25T05:05:36.064 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-25T05:05:36.064 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-25T05:05:36.064 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/93294/exe 2023-11-25T05:05:36.064 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:36.064 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:36.064 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:36.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:36.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:36.065 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:36.065 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:36.065 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:36.065 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:36.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:36.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.065 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:36.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:36.065 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:36.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:36.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:36.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:36.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:36.066 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:36.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:36.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:36.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:36.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:36.067 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:36.067 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:36.067 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:36.067 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:36.067 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:36.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:36.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:36.067 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:36.067 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:36.067 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:36.067 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:36.067 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:36.067 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:36.067 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2023-11-25T05:05:36.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:36.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:36.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:36.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:36.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:36.068 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_cls_remote_reads 2023-11-25T05:05:36.068 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:36.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:36.068 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:36.068 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:36.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:36.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:36.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:36.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:36.069 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:36.069 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:36.069 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:36.069 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:36.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:36.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:36.069 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:36.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:36.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:36.069 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:36.069 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/93372/exe 2023-11-25T05:05:36.070 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:36.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:36.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:36.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:36.070 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2023-11-25T05:05:36.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:36.070 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:36.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:36.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:36.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:36.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:36.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:36.070 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:36.071 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:36.071 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:36.071 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:36.071 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:36.071 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ delete_pools_parallel: /' 2023-11-25T05:05:36.071 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2023-11-25T05:05:36.071 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_delete_pools_parallel 2023-11-25T05:05:36.071 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:36.071 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:36.071 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:36.071 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:36.071 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:36.071 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:36.071 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:36.072 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:36.072 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:36.072 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:36.072 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:36.072 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:36.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:36.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:36.072 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:36.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:36.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:36.073 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:36.073 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-25T05:05:36.073 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-25T05:05:36.073 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-25T05:05:36.073 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/93345/exe 2023-11-25T05:05:36.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-25T05:05:36.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-25T05:05:36.073 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-25T05:05:36.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-25T05:05:36.074 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.074 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2023-11-25T05:05:36.074 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.074 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-25T05:05:36.074 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.074 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2023-11-25T05:05:36.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2023-11-25T05:05:36.074 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:36.074 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2023-11-25T05:05:36.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:36.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:36.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:36.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-25T05:05:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-25T05:05:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-25T05:05:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_open_pools_parallel 2023-11-25T05:05:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-25T05:05:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2023-11-25T05:05:36.076 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ open_pools_parallel: /' 2023-11-25T05:05:36.076 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-25T05:05:36.076 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2023-11-25T05:05:36.076 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2023-11-25T05:05:36.076 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-25T05:05:36.076 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-25T05:05:36.076 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:36.076 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2023-11-25T05:05:36.076 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2023-11-25T05:05:36.076 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2023-11-25T05:05:36.076 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-25T05:05:36.076 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2023-11-25T05:05:36.076 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.077 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-25T05:05:36.077 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.077 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2023-11-25T05:05:36.077 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-25T05:05:36.077 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2023-11-25T05:05:36.077 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-25T05:05:36.077 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2023-11-25T05:05:36.077 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2023-11-25T05:05:36.077 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-25T05:05:36.077 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/93333/exe 2023-11-25T05:05:36.077 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2023-11-25T05:05:36.077 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2023-11-25T05:05:36.077 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2023-11-25T05:05:36.077 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2023-11-25T05:05:36.078 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2023-11-25T05:05:36.078 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2023-11-25T05:05:36.078 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2023-11-25T05:05:36.078 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-25T05:05:36.078 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2023-11-25T05:05:36.078 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-25T05:05:36.078 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2023-11-25T05:05:36.078 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_list_parallel 2023-11-25T05:05:36.078 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ list_parallel: /' 2023-11-25T05:05:36.078 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_list_parallel.log 2023-11-25T05:05:37.409 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faec4958780 1 librados: init done 2023-11-25T05:05:37.409 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.710+0000 7faec4958780 10 librados: wait_for_osdmap waiting 2023-11-25T05:05:37.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.806+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7faeb80157e0 con 0x55f7e2a4b7e0 2023-11-25T05:05:37.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.808+0000 7faeb37fe640 1 --2- 172.21.15.7:0/3026089490 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7faea406aa40 0x7faea406cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:05:37.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.809+0000 7faec245c640 1 --2- 172.21.15.7:0/3026089490 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7faea406aa40 0x7faea406cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:05:37.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.809+0000 7faec245c640 10 monclient: get_auth_request con 0x7faea406aa40 auth_method 0 2023-11-25T05:05:37.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.809+0000 7faec245c640 1 --2- 172.21.15.7:0/3026089490 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7faea406aa40 0x7faea406cf00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faeb400a680 tx=0x7faeb400c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:05:37.411 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.809+0000 7faeb37fe640 10 client.4355.objecter ms_handle_connect 0x7faea406aa40 2023-11-25T05:05:37.411 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.813+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7faeb80576d0 con 0x55f7e2a4b7e0 2023-11-25T05:05:37.411 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.813+0000 7faeb37fe640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_map(22..22 src has 1..22) v4 2023-11-25T05:05:37.411 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.813+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map got epochs [22,22] > 0 2023-11-25T05:05:37.411 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.813+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding full epoch 22 2023-11-25T05:05:37.411 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.813+0000 7faeb37fe640 20 client.4355.objecter dump_active .. 0 homeless 2023-11-25T05:05:37.412 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.813+0000 7faec4958780 10 librados: wait_for_osdmap done waiting 2023-11-25T05:05:37.412 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.813+0000 7faec4958780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:05:37.412 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.813+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55f7e292d760 con 0x55f7e2a4b7e0 2023-11-25T05:05:37.412 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.961+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7faeb8053310 con 0x55f7e2a4b7e0 2023-11-25T05:05:37.412 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.961+0000 7faeb37fe640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-11-25T05:05:37.412 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.961+0000 7faeb3fff640 10 client.4355.objecter _get_latest_version latest 22, have it 2023-11-25T05:05:37.413 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.961+0000 7faec4958780 10 client.4355.objecter create_pool name=ceph_test_rados_api_asio 2023-11-25T05:05:37.413 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.961+0000 7faec4958780 10 client.4355.objecter pool_op_submit 1 2023-11-25T05:05:37.413 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.961+0000 7faec4958780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:05:37.413 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:35.961+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55f7e293e810 con 0x55f7e2a4b7e0 2023-11-25T05:05:37.470 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.407+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== mon.2 v2:172.21.15.7:3301/0 7 ==== pool_op_reply(tid 1 (0) Success v23) v1 ==== 43+0+0 (secure 0 0 0) 0x7faeb805be60 con 0x55f7e2a4b7e0 2023-11-25T05:05:37.470 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.408+0000 7faeb37fe640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 pool_op_reply(tid 1 (0) Success v23) v1 2023-11-25T05:05:37.471 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.408+0000 7faeb37fe640 10 client.4355.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v23) v1 2023-11-25T05:05:37.471 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.408+0000 7faeb37fe640 10 client.4355.objecter have request 1 at 0x55f7e2ae56e0 Op: create 2023-11-25T05:05:37.471 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.408+0000 7faeb37fe640 20 client.4355.objecter waiting for client to reach epoch 23 before calling back 2023-11-25T05:05:37.471 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.408+0000 7faeb37fe640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:05:37.471 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.408+0000 7faeb37fe640 10 monclient: _renew_subs 2023-11-25T05:05:37.472 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.408+0000 7faeb37fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:05:37.472 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.408+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=23}) v3 -- 0x7faea4070780 con 0x55f7e2a4b7e0 2023-11-25T05:05:37.472 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.408+0000 7faeb37fe640 10 client.4355.objecter done 2023-11-25T05:05:37.472 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== mon.2 v2:172.21.15.7:3301/0 8 ==== osd_map(23..23 src has 1..23) v4 ==== 8374+0+0 (secure 0 0 0) 0x7faeb8056fd0 con 0x55f7e2a4b7e0 2023-11-25T05:05:37.473 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faeb37fe640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_map(23..23 src has 1..23) v4 2023-11-25T05:05:37.473 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map got epochs [23,23] > 22 2023-11-25T05:05:37.473 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 23 2023-11-25T05:05:37.473 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faeb37fe640 20 client.4355.objecter dump_active .. 0 homeless 2023-11-25T05:05:37.473 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faec4958780 10 librados: set snap read head -> 1 2023-11-25T05:05:37.473 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faec4958780 10 librados: writefull oid=exist nspace= 2023-11-25T05:05:37.474 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2ae41e0 2023-11-25T05:05:37.474 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 23 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:37.474 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faec4958780 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:37.474 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:37.474 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faec4958780 1 --2- 172.21.15.7:0/3026089490 >> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] conn(0x55f7e2aecea0 0x55f7e2aed280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:05:37.475 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:37.475 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faec4958780 10 client.4355.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 2 osd.7 2023-11-25T05:05:37.475 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:37.475 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 2 2023-11-25T05:05:37.476 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faec4958780 15 client.4355.objecter _send_op 2 to 9.1 on osd.7 2023-11-25T05:05:45.671 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:2 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) v8 -- 0x55f7e2ae5990 con 0x55f7e2aecea0 2023-11-25T05:05:45.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:45.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:37.468+0000 7faec4958780 5 client.4355.objecter 1 in flight 2023-11-25T05:05:45.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:38.496+0000 7faec1c5b640 1 --2- 172.21.15.7:0/3026089490 >> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] conn(0x55f7e2aecea0 0x55f7e2aed280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:05:45.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:38.501+0000 7faec1c5b640 10 monclient: get_auth_request con 0x55f7e2aecea0 auth_method 0 2023-11-25T05:05:45.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:38.577+0000 7faeb37fe640 10 client.4355.objecter ms_handle_connect 0x55f7e2aecea0 2023-11-25T05:05:45.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:38.577+0000 7faec1c5b640 1 --2- 172.21.15.7:0/3026089490 >> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] conn(0x55f7e2aecea0 0x55f7e2aed280 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:05:45.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:38.668+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:45.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:38.668+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:45.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:38.668+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:45.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:38.668+0000 7faeb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:05:45.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:39.669+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:45.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:39.669+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:45.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:39.669+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:45.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:40.669+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:45.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:40.669+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:45.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:40.669+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:45.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:40.711+0000 7faec46c0640 10 client.4355.objecter tick 2023-11-25T05:05:45.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:41.669+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:45.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:41.669+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:45.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:41.669+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:45.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:42.670+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:45.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:42.670+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:45.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:42.670+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:45.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:43.670+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:45.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:43.670+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:45.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:43.670+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:45.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:44.670+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:45.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:44.670+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:45.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:44.670+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:45.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:45.670+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:48.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:45.670+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:48.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:45.670+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:48.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:45.712+0000 7faec46c0640 10 client.4355.objecter tick 2023-11-25T05:05:48.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:46.670+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:48.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:46.670+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:48.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:46.670+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:48.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.374+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 1 ==== osd_map(24..27 src has 1..27) v4 ==== 13419+0+0 (crc 0 0 0) 0x7faebc053900 con 0x55f7e2aecea0 2023-11-25T05:05:48.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.374+0000 7faeb37fe640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_map(24..27 src has 1..27) v4 2023-11-25T05:05:48.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.374+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map got epochs [24,27] > 23 2023-11-25T05:05:48.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.374+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 24 2023-11-25T05:05:48.672 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 10 client.4355.objecter checking op 2 2023-11-25T05:05:48.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 24 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:48.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:48.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 25 2023-11-25T05:05:48.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 10 client.4355.objecter checking op 2 2023-11-25T05:05:48.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 25 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:48.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:48.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 26 2023-11-25T05:05:48.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 10 client.4355.objecter checking op 2 2023-11-25T05:05:48.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 26 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:48.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:48.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 27 2023-11-25T05:05:48.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 10 client.4355.objecter checking op 2 2023-11-25T05:05:48.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 27 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:48.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:48.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 20 client.4355.objecter dump_active .. 0 homeless 2023-11-25T05:05:48.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.375+0000 7faeb37fe640 20 client.4355.objecter 2 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2023-11-25T05:05:48.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.670+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:48.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.670+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:48.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:47.670+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:48.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:48.670+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:48.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:48.670+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:50.719 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:48.670+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:50.719 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:48.670+0000 7faeb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:05:50.719 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:49.670+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:50.719 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:49.670+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:50.719 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:49.670+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:50.719 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.670+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:50.719 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.670+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:50.719 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.670+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:50.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.713+0000 7faec46c0640 10 client.4355.objecter tick 2023-11-25T05:05:50.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.713+0000 7faec46c0640 2 client.4355.objecter tid 2 on osd.7 is laggy 2023-11-25T05:05:50.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.713+0000 7faec46c0640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:05:50.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.713+0000 7faec46c0640 10 monclient: _renew_subs 2023-11-25T05:05:50.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.713+0000 7faec46c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:05:50.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.713+0000 7faec46c0640 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=28}) v3 -- 0x7fae94002f90 con 0x55f7e2a4b7e0 2023-11-25T05:05:50.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.713+0000 7faec46c0640 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- ping magic: 0 v1 -- 0x7fae94003320 con 0x55f7e2aecea0 2023-11-25T05:05:50.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== mon.2 v2:172.21.15.7:3301/0 9 ==== osd_map(28..32 src has 1..32) v4 ==== 3877+0+0 (secure 0 0 0) 0x7faeb8055110 con 0x55f7e2a4b7e0 2023-11-25T05:05:50.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_map(28..32 src has 1..32) v4 2023-11-25T05:05:50.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map got epochs [28,32] > 27 2023-11-25T05:05:50.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 28 2023-11-25T05:05:50.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 10 client.4355.objecter checking op 2 2023-11-25T05:05:50.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 28 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:50.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:50.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 29 2023-11-25T05:05:50.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 10 client.4355.objecter checking op 2 2023-11-25T05:05:50.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 29 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:50.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:50.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 30 2023-11-25T05:05:50.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 10 client.4355.objecter checking op 2 2023-11-25T05:05:50.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 30 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:52.222 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:52.222 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 31 2023-11-25T05:05:52.222 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 10 client.4355.objecter checking op 2 2023-11-25T05:05:52.222 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 31 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:52.222 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:52.222 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 32 2023-11-25T05:05:52.222 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 10 client.4355.objecter checking op 2 2023-11-25T05:05:52.222 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 32 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:52.223 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:52.223 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 20 client.4355.objecter dump_active .. 0 homeless 2023-11-25T05:05:52.223 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:50.717+0000 7faeb37fe640 20 client.4355.objecter 2 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2023-11-25T05:05:52.223 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:51.670+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:52.223 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:51.670+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:52.223 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:51.670+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:52.223 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 2 ==== osd_op_reply(2 exist [writefull 0~5] v27'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7faebc053b60 con 0x55f7e2aecea0 2023-11-25T05:05:52.223 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(2 exist [writefull 0~5] v27'1 uv1 ondisk = 0) v8 2023-11-25T05:05:52.223 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:52.223 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 2 ondisk uv 1 in 9.1 attempt 0 2023-11-25T05:05:52.223 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 0 2023-11-25T05:05:52.224 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 2 2023-11-25T05:05:52.224 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec1c5b640 15 client.4355.objecter _finish_op 2 2023-11-25T05:05:52.224 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:52.224 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 2 2023-11-25T05:05:52.224 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec1c5b640 5 client.4355.objecter 0 in flight 2023-11-25T05:05:52.224 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 10 librados: Objecter returned from writefull r=0 2023-11-25T05:05:52.224 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2023-11-25T05:05:52.224 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a46e80 2023-11-25T05:05:52.224 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 32 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:52.224 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:52.225 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:52.225 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:53.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 10 client.4355.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 3 osd.7 2023-11-25T05:05:53.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:53.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 3 2023-11-25T05:05:53.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 15 client.4355.objecter _send_op 3 to 9.1 on osd.7 2023-11-25T05:05:53.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:3 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f7e2ae4340 con 0x55f7e2aecea0 2023-11-25T05:05:53.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:53.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 5 client.4355.objecter 1 in flight 2023-11-25T05:05:53.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a47320 2023-11-25T05:05:53.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 32 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:53.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 20 client.4355.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2023-11-25T05:05:53.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:53.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:53.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 10 client.4355.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 4 osd.7 2023-11-25T05:05:53.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:53.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 4 2023-11-25T05:05:53.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 15 client.4355.objecter _send_op 4 to 9.1 on osd.7 2023-11-25T05:05:53.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:4 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f7e2a477c0 con 0x55f7e2aecea0 2023-11-25T05:05:53.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 5 2023-11-25T05:05:53.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.215+0000 7faec4958780 5 client.4355.objecter 2 in flight 2023-11-25T05:05:53.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.670+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:53.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.670+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:53.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:52.670+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:53.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 3 ==== osd_map(33..33 src has 1..33) v4 ==== 1020+0+0 (crc 0 0 0) 0x7faebc053db0 con 0x55f7e2aecea0 2023-11-25T05:05:53.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 4 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:53.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faeb37fe640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_map(33..33 src has 1..33) v4 2023-11-25T05:05:53.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-11-25T05:05:53.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:53.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map got epochs [33,33] > 32 2023-11-25T05:05:53.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 33 2023-11-25T05:05:53.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faeb37fe640 10 client.4355.objecter checking op 3 2023-11-25T05:05:53.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 33 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:53.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faeb37fe640 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:53.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faeb37fe640 10 client.4355.objecter checking op 4 2023-11-25T05:05:53.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 33 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:53.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faeb37fe640 20 client.4355.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2023-11-25T05:05:53.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faeb37fe640 20 client.4355.objecter dump_active .. 0 homeless 2023-11-25T05:05:53.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faeb37fe640 20 client.4355.objecter 3 9.d3a59201 osd.7 exist [read 0~256] 2023-11-25T05:05:53.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faeb37fe640 20 client.4355.objecter 4 9.a7bbaa1 osd.7 noexist [read 0~256] 2023-11-25T05:05:53.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 3 ondisk uv 1 in 9.1 attempt 0 2023-11-25T05:05:53.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 5 2023-11-25T05:05:53.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 3 2023-11-25T05:05:53.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 15 client.4355.objecter _finish_op 3 2023-11-25T05:05:53.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 5 2023-11-25T05:05:53.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 3 2023-11-25T05:05:53.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 5 client.4355.objecter 1 in flight 2023-11-25T05:05:53.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 5 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:53.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-11-25T05:05:53.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:53.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 4 ondisk uv 0 in 9.1 attempt 0 2023-11-25T05:05:53.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 10 client.4355.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-11-25T05:05:53.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 0 2023-11-25T05:05:53.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 4 2023-11-25T05:05:53.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 15 client.4355.objecter _finish_op 4 2023-11-25T05:05:53.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:53.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 4 2023-11-25T05:05:53.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec1c5b640 5 client.4355.objecter 0 in flight 2023-11-25T05:05:53.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (803 ms) 2023-11-25T05:05:53.505 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2023-11-25T05:05:53.505 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a46e80 2023-11-25T05:05:53.505 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 33 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:53.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:53.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:53.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:53.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 10 client.4355.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 5 osd.7 2023-11-25T05:05:53.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:53.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 5 2023-11-25T05:05:53.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 15 client.4355.objecter _send_op 5 to 9.1 on osd.7 2023-11-25T05:05:53.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:5 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v8 -- 0x55f7e2a47b20 con 0x55f7e2aecea0 2023-11-25T05:05:53.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:53.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 5 client.4355.objecter 1 in flight 2023-11-25T05:05:53.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a47320 2023-11-25T05:05:53.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 33 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:53.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 20 client.4355.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2023-11-25T05:05:53.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:53.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:53.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 10 client.4355.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 6 osd.7 2023-11-25T05:05:53.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:53.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 6 2023-11-25T05:05:53.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 15 client.4355.objecter _send_op 6 to 9.1 on osd.7 2023-11-25T05:05:53.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:6 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v8 -- 0x55f7e2af5bd0 con 0x55f7e2aecea0 2023-11-25T05:05:53.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 5 2023-11-25T05:05:53.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.018+0000 7faec4958780 5 client.4355.objecter 2 in flight 2023-11-25T05:05:53.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 7 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:53.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 6 ==== osd_map(34..34 src has 1..34) v4 ==== 304+0+0 (crc 0 0 0) 0x7faebc05d420 con 0x55f7e2aecea0 2023-11-25T05:05:53.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faeb37fe640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_map(34..34 src has 1..34) v4 2023-11-25T05:05:53.510 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map got epochs [34,34] > 33 2023-11-25T05:05:53.510 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 34 2023-11-25T05:05:53.510 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faeb37fe640 10 client.4355.objecter checking op 5 2023-11-25T05:05:53.510 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 34 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:53.510 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faeb37fe640 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:53.510 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faeb37fe640 10 client.4355.objecter checking op 6 2023-11-25T05:05:53.510 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 34 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:53.511 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faeb37fe640 20 client.4355.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2023-11-25T05:05:53.511 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faeb37fe640 20 client.4355.objecter dump_active .. 0 homeless 2023-11-25T05:05:53.511 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faeb37fe640 20 client.4355.objecter 5 9.d3a59201 osd.7 exist [read 0~256] 2023-11-25T05:05:53.511 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faeb37fe640 20 client.4355.objecter 6 9.a7bbaa1 osd.7 noexist [read 0~256] 2023-11-25T05:05:53.511 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (485 ms) 2023-11-25T05:05:53.512 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2023-11-25T05:05:53.512 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-11-25T05:05:53.512 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:53.512 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 5 ondisk uv 1 in 9.1 attempt 0 2023-11-25T05:05:53.512 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 5 2023-11-25T05:05:53.512 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 5 2023-11-25T05:05:53.513 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 15 client.4355.objecter _finish_op 5 2023-11-25T05:05:53.513 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 5 2023-11-25T05:05:53.513 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 5 2023-11-25T05:05:53.513 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 5 client.4355.objecter 1 in flight 2023-11-25T05:05:53.513 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 8 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:53.513 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-11-25T05:05:53.513 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:53.514 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 6 ondisk uv 0 in 9.1 attempt 0 2023-11-25T05:05:53.514 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 10 client.4355.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-11-25T05:05:53.514 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 0 2023-11-25T05:05:53.514 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 6 2023-11-25T05:05:53.515 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 15 client.4355.objecter _finish_op 6 2023-11-25T05:05:53.515 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:53.515 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 6 2023-11-25T05:05:53.515 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec1c5b640 5 client.4355.objecter 0 in flight 2023-11-25T05:05:53.516 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2b15f40 2023-11-25T05:05:53.516 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 34 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:53.516 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:53.516 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:53.516 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:53.516 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 10 client.4355.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 7 osd.7 2023-11-25T05:05:53.516 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:53.517 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 7 2023-11-25T05:05:53.517 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 15 client.4355.objecter _send_op 7 to 9.1 on osd.7 2023-11-25T05:05:53.517 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:7 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v8 -- 0x55f7e2b163e0 con 0x55f7e2aecea0 2023-11-25T05:05:53.517 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:53.517 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 5 client.4355.objecter 1 in flight 2023-11-25T05:05:53.517 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2b36860 2023-11-25T05:05:53.517 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 34 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:53.517 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 20 client.4355.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2023-11-25T05:05:53.518 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:53.518 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:53.518 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 10 client.4355.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 8 osd.7 2023-11-25T05:05:53.519 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:53.519 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 8 2023-11-25T05:05:53.519 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 15 client.4355.objecter _send_op 8 to 9.1 on osd.7 2023-11-25T05:05:53.519 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:8 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v8 -- 0x55f7e2a25d50 con 0x55f7e2aecea0 2023-11-25T05:05:53.519 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 5 2023-11-25T05:05:53.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.503+0000 7faec4958780 5 client.4355.objecter 2 in flight 2023-11-25T05:05:53.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.514+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 9 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:53.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.514+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-11-25T05:05:53.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.514+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:53.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.514+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 7 ondisk uv 1 in 9.1 attempt 0 2023-11-25T05:05:53.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.514+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 5 2023-11-25T05:05:53.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.514+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 7 2023-11-25T05:05:53.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.514+0000 7faec1c5b640 15 client.4355.objecter _finish_op 7 2023-11-25T05:05:53.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.514+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 5 2023-11-25T05:05:53.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.514+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 7 2023-11-25T05:05:53.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.514+0000 7faec1c5b640 5 client.4355.objecter 1 in flight 2023-11-25T05:05:53.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (14 ms) 2023-11-25T05:05:53.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2023-11-25T05:05:53.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 10 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:53.528 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-11-25T05:05:53.528 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:53.528 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 8 ondisk uv 0 in 9.1 attempt 0 2023-11-25T05:05:53.528 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec1c5b640 10 client.4355.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-11-25T05:05:53.528 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 0 2023-11-25T05:05:53.528 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 8 2023-11-25T05:05:53.529 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec1c5b640 15 client.4355.objecter _finish_op 8 2023-11-25T05:05:53.529 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:53.529 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 8 2023-11-25T05:05:53.529 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec1c5b640 5 client.4355.objecter 0 in flight 2023-11-25T05:05:53.529 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec4958780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-11-25T05:05:53.529 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec4958780 20 librados: queue_aio_write 0x55f7e292d480 completion 0x55f7e2b36860 write_seq 1 2023-11-25T05:05:53.530 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a46e80 2023-11-25T05:05:53.530 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 34 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:53.530 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec4958780 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:53.530 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:53.954 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:36.152 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:53.963 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:36.161 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:54.015 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:36.159 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:54.017 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:36.161 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:54.047 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:54.047 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec4958780 10 client.4355.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 9 osd.7 2023-11-25T05:05:54.047 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:54.047 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 9 2023-11-25T05:05:54.047 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec4958780 15 client.4355.objecter _send_op 9 to 9.1 on osd.7 2023-11-25T05:05:54.047 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:9 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34) v8 -- 0x55f7e2b36970 con 0x55f7e2aecea0 2023-11-25T05:05:54.047 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:54.047 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec4958780 5 client.4355.objecter 1 in flight 2023-11-25T05:05:54.047 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.517+0000 7faec4958780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-11-25T05:05:54.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.670+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:54.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.670+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:54.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:53.670+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:54.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 11 ==== osd_op_reply(9 exist [write 0~5] v34'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:54.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(9 exist [write 0~5] v34'2 uv2 ondisk = 0) v8 2023-11-25T05:05:54.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:54.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 9 ondisk uv 2 in 9.1 attempt 0 2023-11-25T05:05:54.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 0 2023-11-25T05:05:54.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 9 2023-11-25T05:05:54.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec1c5b640 15 client.4355.objecter _finish_op 9 2023-11-25T05:05:54.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:54.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 9 2023-11-25T05:05:54.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec1c5b640 5 client.4355.objecter 0 in flight 2023-11-25T05:05:54.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec1c5b640 20 librados: complete_aio_write 0x55f7e2b36860 2023-11-25T05:05:54.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (528 ms) 2023-11-25T05:05:54.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2023-11-25T05:05:54.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec4958780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-11-25T05:05:54.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec4958780 20 librados: queue_aio_write 0x55f7e292d480 completion 0x55f7e2a47320 write_seq 2 2023-11-25T05:05:54.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a46e80 2023-11-25T05:05:54.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 34 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:54.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec4958780 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:54.195 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:36.393 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:54.196 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:36.339 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:54.200 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:36.398 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:54.362 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:36.559 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:54.407 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:36.605 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:05:55.002 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:55.002 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:55.003 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec4958780 10 client.4355.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 10 osd.7 2023-11-25T05:05:55.003 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:55.003 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 10 2023-11-25T05:05:55.003 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec4958780 15 client.4355.objecter _send_op 10 to 9.1 on osd.7 2023-11-25T05:05:55.003 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:10 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34) v8 -- 0x55f7e2b36970 con 0x55f7e2aecea0 2023-11-25T05:05:55.003 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:55.003 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec4958780 5 client.4355.objecter 1 in flight 2023-11-25T05:05:55.003 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.045+0000 7faec4958780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-11-25T05:05:55.003 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.671+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:55.003 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.671+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:55.003 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:54.671+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:55.003 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.000+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 12 ==== osd_op_reply(10 exist [write 0~5] v34'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:55.004 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.000+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(10 exist [write 0~5] v34'3 uv3 ondisk = 0) v8 2023-11-25T05:05:55.004 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.000+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:55.004 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.000+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 10 ondisk uv 3 in 9.1 attempt 0 2023-11-25T05:05:55.004 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.000+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 0 2023-11-25T05:05:55.004 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.000+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 10 2023-11-25T05:05:55.004 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.000+0000 7faec1c5b640 15 client.4355.objecter _finish_op 10 2023-11-25T05:05:55.004 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:55.004 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 10 2023-11-25T05:05:55.004 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec1c5b640 5 client.4355.objecter 0 in flight 2023-11-25T05:05:55.004 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec1c5b640 20 librados: complete_aio_write 0x55f7e2a47320 2023-11-25T05:05:55.004 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (955 ms) 2023-11-25T05:05:55.004 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2023-11-25T05:05:55.005 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec4958780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-11-25T05:05:55.005 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec4958780 20 librados: queue_aio_write 0x55f7e292d480 completion 0x55f7e2b36b90 write_seq 3 2023-11-25T05:05:55.005 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a46e80 2023-11-25T05:05:55.005 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 34 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:56.076 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec4958780 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:56.076 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:56.076 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:56.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec4958780 10 client.4355.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 11 osd.7 2023-11-25T05:05:56.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:56.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 11 2023-11-25T05:05:56.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec4958780 15 client.4355.objecter _send_op 11 to 9.1 on osd.7 2023-11-25T05:05:56.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:11 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34) v8 -- 0x55f7e2a29830 con 0x55f7e2aecea0 2023-11-25T05:05:56.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:56.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec4958780 5 client.4355.objecter 1 in flight 2023-11-25T05:05:56.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.001+0000 7faec4958780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-11-25T05:05:56.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.016+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 13 ==== osd_map(35..35 src has 1..35) v4 ==== 675+0+0 (crc 0 0 0) 0x7faebc05d5e0 con 0x55f7e2aecea0 2023-11-25T05:05:56.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.016+0000 7faeb37fe640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_map(35..35 src has 1..35) v4 2023-11-25T05:05:56.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.016+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map got epochs [35,35] > 34 2023-11-25T05:05:56.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.016+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 35 2023-11-25T05:05:56.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.016+0000 7faeb37fe640 10 client.4355.objecter checking op 11 2023-11-25T05:05:56.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.016+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 35 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:56.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.016+0000 7faeb37fe640 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:56.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.016+0000 7faeb37fe640 20 client.4355.objecter dump_active .. 0 homeless 2023-11-25T05:05:56.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.016+0000 7faeb37fe640 20 client.4355.objecter 11 9.d3a59201 osd.7 exist [write 0~5 in=5b] 2023-11-25T05:05:56.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.671+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:56.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.671+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:56.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.671+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:56.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:55.714+0000 7faec46c0640 10 client.4355.objecter tick 2023-11-25T05:05:56.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 14 ==== osd_op_reply(11 exist [write 0~5] v35'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:56.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(11 exist [write 0~5] v35'4 uv4 ondisk = 0) v8 2023-11-25T05:05:56.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:56.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 11 ondisk uv 4 in 9.1 attempt 0 2023-11-25T05:05:56.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 0 2023-11-25T05:05:56.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 11 2023-11-25T05:05:56.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec1c5b640 15 client.4355.objecter _finish_op 11 2023-11-25T05:05:56.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:56.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 11 2023-11-25T05:05:56.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec1c5b640 5 client.4355.objecter 0 in flight 2023-11-25T05:05:56.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec1c5b640 20 librados: complete_aio_write 0x55f7e2b36b90 2023-11-25T05:05:56.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (1073 ms) 2023-11-25T05:05:56.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2023-11-25T05:05:56.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a46e80 2023-11-25T05:05:56.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 35 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:56.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:56.080 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:56.080 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:56.080 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 10 client.4355.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 12 osd.7 2023-11-25T05:05:56.080 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:56.080 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 12 2023-11-25T05:05:56.081 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 15 client.4355.objecter _send_op 12 to 9.1 on osd.7 2023-11-25T05:05:56.081 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:12 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35) v8 -- 0x55f7e2b16070 con 0x55f7e2aecea0 2023-11-25T05:05:56.081 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:56.081 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 5 client.4355.objecter 1 in flight 2023-11-25T05:05:56.081 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a292d0 2023-11-25T05:05:56.082 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 35 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:56.082 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 20 client.4355.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2023-11-25T05:05:56.082 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:56.082 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:56.082 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 10 client.4355.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 13 osd.7 2023-11-25T05:05:56.082 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:56.082 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 13 2023-11-25T05:05:56.082 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 15 client.4355.objecter _send_op 13 to 9.1 on osd.7 2023-11-25T05:05:56.101 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:13 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35) v8 -- 0x55f7e2a29770 con 0x55f7e2aecea0 2023-11-25T05:05:56.101 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 5 2023-11-25T05:05:56.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.074+0000 7faec4958780 5 client.4355.objecter 2 in flight 2023-11-25T05:05:56.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.091+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 15 ==== osd_map(36..36 src has 1..36) v4 ==== 667+0+0 (crc 0 0 0) 0x7faebc05daa0 con 0x55f7e2aecea0 2023-11-25T05:05:56.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.091+0000 7faeb37fe640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_map(36..36 src has 1..36) v4 2023-11-25T05:05:56.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.091+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map got epochs [36,36] > 35 2023-11-25T05:05:56.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.091+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 36 2023-11-25T05:05:56.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.091+0000 7faeb37fe640 10 client.4355.objecter checking op 12 2023-11-25T05:05:56.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.091+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 36 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:56.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.091+0000 7faeb37fe640 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:56.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.091+0000 7faeb37fe640 10 client.4355.objecter checking op 13 2023-11-25T05:05:56.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.091+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 36 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:56.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.091+0000 7faeb37fe640 20 client.4355.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2023-11-25T05:05:56.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.091+0000 7faeb37fe640 20 client.4355.objecter dump_active .. 0 homeless 2023-11-25T05:05:56.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.091+0000 7faeb37fe640 20 client.4355.objecter 12 9.d3a59201 osd.7 exist [read 0~0] 2023-11-25T05:05:56.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.091+0000 7faeb37fe640 20 client.4355.objecter 13 9.a7bbaa1 osd.7 noexist [read 0~0] 2023-11-25T05:05:56.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.099+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 16 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:56.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-11-25T05:05:56.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:56.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 12 ondisk uv 4 in 9.1 attempt 0 2023-11-25T05:05:56.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 5 2023-11-25T05:05:56.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 12 2023-11-25T05:05:56.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 15 client.4355.objecter _finish_op 12 2023-11-25T05:05:56.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 5 2023-11-25T05:05:56.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 12 2023-11-25T05:05:56.104 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 5 client.4355.objecter 1 in flight 2023-11-25T05:05:56.104 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 17 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:56.104 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-11-25T05:05:56.104 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:56.104 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 13 ondisk uv 0 in 9.1 attempt 0 2023-11-25T05:05:56.104 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 10 client.4355.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-11-25T05:05:56.104 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 0 2023-11-25T05:05:56.104 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 13 2023-11-25T05:05:56.104 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 15 client.4355.objecter _finish_op 13 2023-11-25T05:05:56.104 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:56.104 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 13 2023-11-25T05:05:56.104 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec1c5b640 5 client.4355.objecter 0 in flight 2023-11-25T05:05:56.104 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (26 ms) 2023-11-25T05:05:56.105 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2023-11-25T05:05:56.105 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a292d0 2023-11-25T05:05:56.105 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 36 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:56.105 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:56.105 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:56.105 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:56.105 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 10 client.4355.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 14 osd.7 2023-11-25T05:05:56.105 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:56.105 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 14 2023-11-25T05:05:56.105 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 15 client.4355.objecter _send_op 14 to 9.1 on osd.7 2023-11-25T05:05:56.105 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:14 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x55f7e2a46f90 con 0x55f7e2aecea0 2023-11-25T05:05:56.105 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:56.105 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 5 client.4355.objecter 1 in flight 2023-11-25T05:05:56.105 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a260b0 2023-11-25T05:05:56.106 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 36 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:56.106 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 20 client.4355.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2023-11-25T05:05:56.106 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:56.106 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:56.116 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 10 client.4355.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 15 osd.7 2023-11-25T05:05:56.116 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:56.116 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 15 2023-11-25T05:05:56.116 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 15 client.4355.objecter _send_op 15 to 9.1 on osd.7 2023-11-25T05:05:56.117 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:15 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x55f7e2a29880 con 0x55f7e2aecea0 2023-11-25T05:05:56.117 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 5 2023-11-25T05:05:56.117 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.100+0000 7faec4958780 5 client.4355.objecter 2 in flight 2023-11-25T05:05:56.117 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 18 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:56.117 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-11-25T05:05:56.117 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:56.117 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 14 ondisk uv 4 in 9.1 attempt 0 2023-11-25T05:05:56.117 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 5 2023-11-25T05:05:56.117 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 14 2023-11-25T05:05:56.117 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 15 client.4355.objecter _finish_op 14 2023-11-25T05:05:56.117 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 5 2023-11-25T05:05:56.117 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 14 2023-11-25T05:05:56.117 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 5 client.4355.objecter 1 in flight 2023-11-25T05:05:56.118 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 19 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:56.118 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-11-25T05:05:56.118 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:56.118 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 15 ondisk uv 0 in 9.1 attempt 0 2023-11-25T05:05:56.118 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 10 client.4355.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-11-25T05:05:56.118 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 0 2023-11-25T05:05:56.118 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 15 2023-11-25T05:05:56.118 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 15 client.4355.objecter _finish_op 15 2023-11-25T05:05:56.118 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:56.118 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 15 2023-11-25T05:05:56.118 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (14 ms) 2023-11-25T05:05:56.128 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.114+0000 7faec1c5b640 5 client.4355.objecter 0 in flight 2023-11-25T05:05:56.128 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2023-11-25T05:05:56.128 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a260b0 2023-11-25T05:05:56.128 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 36 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:56.128 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:56.128 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:56.128 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:56.128 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 10 client.4355.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 16 osd.7 2023-11-25T05:05:56.128 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:56.128 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 16 2023-11-25T05:05:56.128 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 15 client.4355.objecter _send_op 16 to 9.1 on osd.7 2023-11-25T05:05:56.129 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:16 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x55f7e2a46f90 con 0x55f7e2aecea0 2023-11-25T05:05:56.129 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:56.129 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 5 client.4355.objecter 1 in flight 2023-11-25T05:05:56.129 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a26550 2023-11-25T05:05:56.129 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 36 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2023-11-25T05:05:56.129 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 20 client.4355.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2023-11-25T05:05:56.129 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:56.129 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:56.129 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 10 client.4355.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 17 osd.7 2023-11-25T05:05:56.129 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:56.129 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 17 2023-11-25T05:05:56.129 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 15 client.4355.objecter _send_op 17 to 9.1 on osd.7 2023-11-25T05:05:56.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:17 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x55f7e2a26a10 con 0x55f7e2aecea0 2023-11-25T05:05:56.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 5 2023-11-25T05:05:56.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.115+0000 7faec4958780 5 client.4355.objecter 2 in flight 2023-11-25T05:05:56.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.126+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 20 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:56.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.126+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-11-25T05:05:56.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.126+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:56.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.126+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 16 ondisk uv 4 in 9.1 attempt 0 2023-11-25T05:05:56.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.126+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 5 2023-11-25T05:05:56.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.126+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 16 2023-11-25T05:05:56.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.126+0000 7faec1c5b640 15 client.4355.objecter _finish_op 16 2023-11-25T05:05:56.132 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.126+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 5 2023-11-25T05:05:56.132 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.126+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 16 2023-11-25T05:05:56.132 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.126+0000 7faec1c5b640 5 client.4355.objecter 1 in flight 2023-11-25T05:05:56.132 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.129+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 21 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:56.132 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.129+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-11-25T05:05:56.132 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.129+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:56.132 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.129+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 17 ondisk uv 0 in 9.1 attempt 0 2023-11-25T05:05:56.132 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.129+0000 7faec1c5b640 10 client.4355.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-11-25T05:05:56.132 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.129+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 0 2023-11-25T05:05:56.132 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.129+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 17 2023-11-25T05:05:56.132 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.129+0000 7faec1c5b640 15 client.4355.objecter _finish_op 17 2023-11-25T05:05:56.133 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.129+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:56.133 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.129+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 17 2023-11-25T05:05:56.133 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.129+0000 7faec1c5b640 5 client.4355.objecter 0 in flight 2023-11-25T05:05:56.133 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (15 ms) 2023-11-25T05:05:56.133 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2023-11-25T05:05:56.133 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.130+0000 7faec4958780 20 librados: queue_aio_write 0x55f7e292d480 completion 0x55f7e2a29ae0 write_seq 4 2023-11-25T05:05:56.133 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.130+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a292d0 2023-11-25T05:05:56.133 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.130+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 36 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:56.133 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.130+0000 7faec4958780 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:56.133 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.130+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:56.133 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.130+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:56.133 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.130+0000 7faec4958780 10 client.4355.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 18 osd.7 2023-11-25T05:05:56.133 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.130+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:57.013 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.130+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 18 2023-11-25T05:05:57.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.130+0000 7faec4958780 15 client.4355.objecter _send_op 18 to 9.1 on osd.7 2023-11-25T05:05:57.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.130+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:18 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) v8 -- 0x55f7e2a46f90 con 0x55f7e2aecea0 2023-11-25T05:05:57.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.130+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:57.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.130+0000 7faec4958780 5 client.4355.objecter 1 in flight 2023-11-25T05:05:57.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.671+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:57.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.671+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:57.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:56.671+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:57.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 22 ==== osd_op_reply(18 exist [writefull 0~5] v36'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:57.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(18 exist [writefull 0~5] v36'5 uv5 ondisk = 0) v8 2023-11-25T05:05:57.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:57.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 18 ondisk uv 5 in 9.1 attempt 0 2023-11-25T05:05:57.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 0 2023-11-25T05:05:57.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 18 2023-11-25T05:05:57.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec1c5b640 15 client.4355.objecter _finish_op 18 2023-11-25T05:05:57.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:57.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 18 2023-11-25T05:05:57.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec1c5b640 5 client.4355.objecter 0 in flight 2023-11-25T05:05:57.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (882 ms) 2023-11-25T05:05:57.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec1c5b640 20 librados: complete_aio_write 0x55f7e2a29ae0 2023-11-25T05:05:57.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2023-11-25T05:05:57.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec4958780 20 librados: queue_aio_write 0x55f7e292d480 completion 0x55f7e2a29770 write_seq 5 2023-11-25T05:05:57.016 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a292d0 2023-11-25T05:05:57.016 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 36 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:57.016 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec4958780 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:57.016 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:57.016 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:57.016 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec4958780 10 client.4355.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 19 osd.7 2023-11-25T05:05:57.016 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:57.016 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 19 2023-11-25T05:05:57.748 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec4958780 15 client.4355.objecter _send_op 19 to 9.1 on osd.7 2023-11-25T05:05:57.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:19 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) v8 -- 0x55f7e2a46f90 con 0x55f7e2aecea0 2023-11-25T05:05:57.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:57.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.011+0000 7faec4958780 5 client.4355.objecter 1 in flight 2023-11-25T05:05:57.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.070+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 23 ==== osd_map(37..37 src has 1..37) v4 ==== 984+0+0 (crc 0 0 0) 0x7faebc072410 con 0x55f7e2aecea0 2023-11-25T05:05:57.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.070+0000 7faeb37fe640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_map(37..37 src has 1..37) v4 2023-11-25T05:05:57.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.070+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map got epochs [37,37] > 36 2023-11-25T05:05:57.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.070+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 37 2023-11-25T05:05:57.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.070+0000 7faeb37fe640 10 client.4355.objecter checking op 19 2023-11-25T05:05:57.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.070+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 37 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:57.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.070+0000 7faeb37fe640 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:57.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.070+0000 7faeb37fe640 20 client.4355.objecter dump_active .. 0 homeless 2023-11-25T05:05:57.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.070+0000 7faeb37fe640 20 client.4355.objecter 19 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2023-11-25T05:05:57.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.671+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:57.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.671+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:57.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.671+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:57.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 24 ==== osd_op_reply(19 exist [writefull 0~5] v37'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:57.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(19 exist [writefull 0~5] v37'6 uv6 ondisk = 0) v8 2023-11-25T05:05:57.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:57.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 19 ondisk uv 6 in 9.1 attempt 0 2023-11-25T05:05:57.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 0 2023-11-25T05:05:57.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 19 2023-11-25T05:05:57.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec1c5b640 15 client.4355.objecter _finish_op 19 2023-11-25T05:05:57.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:57.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 19 2023-11-25T05:05:57.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec1c5b640 5 client.4355.objecter 0 in flight 2023-11-25T05:05:57.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec1c5b640 20 librados: complete_aio_write 0x55f7e2a29770 2023-11-25T05:05:57.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (736 ms) 2023-11-25T05:05:57.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2023-11-25T05:05:58.784 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec4958780 20 librados: queue_aio_write 0x55f7e292d480 completion 0x55f7e2a26df0 write_seq 6 2023-11-25T05:05:58.784 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec4958780 10 client.4355.objecter _op_submit op 0x55f7e2a260b0 2023-11-25T05:05:58.784 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec4958780 20 client.4355.objecter _calc_target epoch 37 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:58.784 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec4958780 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:58.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec4958780 10 client.4355.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2023-11-25T05:05:58.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec4958780 20 client.4355.objecter _get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:58.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec4958780 10 client.4355.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 20 osd.7 2023-11-25T05:05:58.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec4958780 20 client.4355.objecter get_session s=0x55f7e28f2140 osd=7 3 2023-11-25T05:05:58.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec4958780 15 client.4355.objecter _session_op_assign 7 20 2023-11-25T05:05:58.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec4958780 15 client.4355.objecter _send_op 20 to 9.1 on osd.7 2023-11-25T05:05:58.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- osd_op(unknown.0.0:20 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) v8 -- 0x55f7e2a46f90 con 0x55f7e2aecea0 2023-11-25T05:05:58.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:58.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:57.747+0000 7faec4958780 5 client.4355.objecter 1 in flight 2023-11-25T05:05:58.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.197+0000 7faeb37fe640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 25 ==== osd_map(38..38 src has 1..38) v4 ==== 1060+0+0 (crc 0 0 0) 0x7faebc072410 con 0x55f7e2aecea0 2023-11-25T05:05:58.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.197+0000 7faeb37fe640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_map(38..38 src has 1..38) v4 2023-11-25T05:05:58.785 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.197+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map got epochs [38,38] > 37 2023-11-25T05:05:58.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.197+0000 7faeb37fe640 3 client.4355.objecter handle_osd_map decoding incremental epoch 38 2023-11-25T05:05:58.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.197+0000 7faeb37fe640 10 client.4355.objecter checking op 20 2023-11-25T05:05:58.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.197+0000 7faeb37fe640 20 client.4355.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2023-11-25T05:05:58.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.197+0000 7faeb37fe640 20 client.4355.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2023-11-25T05:05:58.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.197+0000 7faeb37fe640 20 client.4355.objecter dump_active .. 0 homeless 2023-11-25T05:05:58.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.197+0000 7faeb37fe640 20 client.4355.objecter 20 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2023-11-25T05:05:58.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.671+0000 7faeb27fc640 10 monclient: tick 2023-11-25T05:05:58.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.671+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2023-11-25T05:05:58.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.671+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:05:58.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.671+0000 7faeb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:05:58.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec1c5b640 1 -- 172.21.15.7:0/3026089490 <== osd.7 v2:172.21.15.90:6802/815268397 26 ==== osd_op_reply(20 exist [writefull 0~5] v38'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7faebc06e020 con 0x55f7e2aecea0 2023-11-25T05:05:58.786 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec1c5b640 10 client.4355.objecter ms_dispatch 0x55f7e29271b0 osd_op_reply(20 exist [writefull 0~5] v38'7 uv7 ondisk = 0) v8 2023-11-25T05:05:58.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec1c5b640 10 client.4355.objecter in handle_osd_op_reply 2023-11-25T05:05:58.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec1c5b640 7 client.4355.objecter handle_osd_op_reply 20 ondisk uv 7 in 9.1 attempt 0 2023-11-25T05:05:58.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec1c5b640 10 client.4355.objecter op 0 rval 0 len 0 2023-11-25T05:05:58.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec1c5b640 15 client.4355.objecter handle_osd_op_reply completed tid 20 2023-11-25T05:05:58.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec1c5b640 15 client.4355.objecter _finish_op 20 2023-11-25T05:05:58.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec1c5b640 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 4 2023-11-25T05:05:58.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec1c5b640 15 client.4355.objecter _session_op_remove 7 20 2023-11-25T05:05:58.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec1c5b640 5 client.4355.objecter 0 in flight 2023-11-25T05:05:58.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec1c5b640 20 librados: complete_aio_write 0x55f7e2a26df0 2023-11-25T05:05:58.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (1035 ms) 2023-11-25T05:05:58.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec4958780 10 librados: watch_flush enter 2023-11-25T05:05:58.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec4958780 10 librados: watch_flush exit 2023-11-25T05:05:58.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec4958780 10 client.4355.objecter close_session for osd.7 2023-11-25T05:05:58.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 >> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] conn(0x55f7e2aecea0 msgr2=0x55f7e2aed280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:05:58.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec4958780 1 --2- 172.21.15.7:0/3026089490 >> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] conn(0x55f7e2aecea0 0x55f7e2aed280 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:05:58.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec4958780 20 client.4355.objecter put_session s=0x55f7e28f2140 osd=7 1 2023-11-25T05:05:58.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec4958780 20 client.4355.objecter shutdown clearing up homeless session... 2023-11-25T05:05:58.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.782+0000 7faec4958780 10 client.4355.objecter successfully canceled tick 2023-11-25T05:05:58.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.783+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7faea406aa40 msgr2=0x7faea406cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:05:58.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.783+0000 7faec4958780 1 --2- 172.21.15.7:0/3026089490 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7faea406aa40 0x7faea406cf00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faeb400a680 tx=0x7faeb400c040 comp rx=0 tx=0).stop 2023-11-25T05:05:58.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.783+0000 7faec4958780 10 monclient: shutdown 2023-11-25T05:05:58.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.783+0000 7faec4958780 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:05:58.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.783+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55f7e2a4b7e0 msgr2=0x55f7e2a44fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:05:58.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.783+0000 7faec4958780 1 --2- 172.21.15.7:0/3026089490 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55f7e2a4b7e0 0x55f7e2a44fc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faeb8009df0 tx=0x7faeb8009ee0 comp rx=0 tx=0).stop 2023-11-25T05:05:58.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.783+0000 7faec2c5d640 1 -- 172.21.15.7:0/3026089490 reap_dead start 2023-11-25T05:05:58.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.783+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 shutdown_connections 2023-11-25T05:05:58.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.783+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 >> 172.21.15.7:0/3026089490 conn(0x55f7e29bebc0 msgr2=0x55f7e29bd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:05:58.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.783+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 shutdown_connections 2023-11-25T05:05:58.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.783+0000 7faec4958780 1 -- 172.21.15.7:0/3026089490 wait complete. 2023-11-25T05:05:58.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.783+0000 7faec4958780 1 librados: shutdown 2023-11-25T05:05:58.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [----------] 12 tests from AsioRados (6567 ms total) 2023-11-25T05:05:58.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2023-11-25T05:05:58.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [----------] Global test environment tear-down 2023-11-25T05:05:58.790 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (23165 ms total) 2023-11-25T05:05:58.790 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ PASSED ] 12 tests. 2023-11-25T05:06:05.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2023-11-25T05:06:05.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [----------] Global test environment set-up. 2023-11-25T05:06:05.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [----------] 4 tests from LibRadosService 2023-11-25T05:06:05.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2023-11-25T05:06:05.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5682 ms) 2023-11-25T05:06:05.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2023-11-25T05:06:05.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ OK ] LibRadosService.RegisterLate (55 ms) 2023-11-25T05:06:05.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2023-11-25T05:06:05.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: cluster: 2023-11-25T05:06:05.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: id: 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:06:05.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: health: HEALTH_WARN 2023-11-25T05:06:05.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 7 pool(s) do not have an application enabled 2023-11-25T05:06:05.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2023-11-25T05:06:05.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: services: 2023-11-25T05:06:05.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-11-25T05:06:05.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mgr: x(active, since 2m), standbys: y 2023-11-25T05:06:05.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: osd: 8 osds: 8 up (since 77s), 8 in (since 2m) 2023-11-25T05:06:05.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: laundry: 2 daemons active (1 hosts) 2023-11-25T05:06:05.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2023-11-25T05:06:05.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: data: 2023-11-25T05:06:05.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pools: 26 pools, 777 pgs 2023-11-25T05:06:05.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: objects: 2 objects, 449 KiB 2023-11-25T05:06:05.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-11-25T05:06:05.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pgs: 98.842% pgs unknown 2023-11-25T05:06:05.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 768 unknown 2023-11-25T05:06:05.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 9 active+clean 2023-11-25T05:06:05.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2023-11-25T05:06:05.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2023-11-25T05:06:05.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: cluster: 2023-11-25T05:06:05.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: id: 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:06:05.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: health: HEALTH_WARN 2023-11-25T05:06:05.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 7 pool(s) do not have an application enabled 2023-11-25T05:06:05.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2023-11-25T05:06:05.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: services: 2023-11-25T05:06:05.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-11-25T05:06:05.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mgr: x(active, since 2m), standbys: y 2023-11-25T05:06:05.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: osd: 8 osds: 8 up (since 80s), 8 in (since 2m) 2023-11-25T05:06:05.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2023-11-25T05:06:05.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-11-25T05:06:05.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2023-11-25T05:06:05.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: data: 2023-11-25T05:06:05.356 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pools: 29 pools, 849 pgs 2023-11-25T05:06:05.356 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: objects: 2 objects, 449 KiB 2023-11-25T05:06:05.356 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-11-25T05:06:05.356 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pgs: 50.059% pgs unknown 2023-11-25T05:06:05.356 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 48.881% pgs not active 2023-11-25T05:06:05.356 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 425 unknown 2023-11-25T05:06:05.356 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 334 creating+peering 2023-11-25T05:06:05.356 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 81 creating+activating 2023-11-25T05:06:05.356 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 9 active+clean 2023-11-25T05:06:05.356 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2023-11-25T05:06:05.356 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2023-11-25T05:06:05.356 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ OK ] LibRadosService.StatusFormat (3421 ms) 2023-11-25T05:06:05.357 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ RUN ] LibRadosService.Status 2023-11-25T05:06:05.357 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ OK ] LibRadosService.Status (20350 ms) 2023-11-25T05:06:05.357 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [----------] 4 tests from LibRadosService (29508 ms total) 2023-11-25T05:06:05.357 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2023-11-25T05:06:05.357 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [----------] Global test environment tear-down 2023-11-25T05:06:05.357 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [==========] 4 tests from 1 test suite ran. (29508 ms total) 2023-11-25T05:06:05.357 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ PASSED ] 4 tests. 2023-11-25T05:06:06.666 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2023-11-25T05:06:06.666 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [----------] Global test environment set-up. 2023-11-25T05:06:06.666 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2023-11-25T05:06:06.666 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2023-11-25T05:06:06.666 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5394 ms) 2023-11-25T05:06:06.666 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2023-11-25T05:06:06.666 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (95 ms) 2023-11-25T05:06:06.666 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2023-11-25T05:06:06.667 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20060 ms) 2023-11-25T05:06:06.667 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2023-11-25T05:06:06.667 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: attempt 0 of 20 2023-11-25T05:06:06.667 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: failed to deregister: 2023-11-25T05:06:06.667 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: { 2023-11-25T05:06:06.667 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "epoch": 12, 2023-11-25T05:06:06.667 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "modified": "2023-11-25T05:06:03.398594+0000", 2023-11-25T05:06:06.667 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "services": { 2023-11-25T05:06:06.667 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "laundry": { 2023-11-25T05:06:06.668 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "daemons": { 2023-11-25T05:06:06.669 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "summary": "", 2023-11-25T05:06:06.669 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "close-test-pid93323": { 2023-11-25T05:06:06.669 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "start_epoch": 12, 2023-11-25T05:06:06.669 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "start_stamp": "2023-11-25T05:06:01.564538+0000", 2023-11-25T05:06:06.669 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "gid": 4729, 2023-11-25T05:06:06.669 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "addr": "172.21.15.7:0/3850763885", 2023-11-25T05:06:06.669 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "metadata": { 2023-11-25T05:06:06.669 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "arch": "x86_64", 2023-11-25T05:06:06.669 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "ceph_release": "reef", 2023-11-25T05:06:06.669 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "ceph_version": "ceph version 18.0.0-7506-gec928368 (ec928368edcc5fc71884a56fd9fd821b148f4e30) reef (dev)", 2023-11-25T05:06:06.669 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "ceph_version_short": "18.0.0-7506-gec928368", 2023-11-25T05:06:06.669 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2023-11-25T05:06:06.669 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "distro": "centos", 2023-11-25T05:06:06.669 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-11-25T05:06:06.670 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "distro_version": "9", 2023-11-25T05:06:06.670 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "foo": "bar", 2023-11-25T05:06:06.670 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "hostname": "smithi007", 2023-11-25T05:06:06.670 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Nov 15 10:38:22 UTC 2023", 2023-11-25T05:06:06.670 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "kernel_version": "5.14.0-386.el9.x86_64", 2023-11-25T05:06:06.670 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "mem_swap_kb": "0", 2023-11-25T05:06:06.670 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "mem_total_kb": "32494892", 2023-11-25T05:06:06.670 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "os": "Linux", 2023-11-25T05:06:06.670 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "this": "that" 2023-11-25T05:06:06.670 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: }, 2023-11-25T05:06:06.670 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "task_status": {} 2023-11-25T05:06:06.670 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: }, 2023-11-25T05:06:06.670 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "pid93301": { 2023-11-25T05:06:06.671 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "start_epoch": 10, 2023-11-25T05:06:06.671 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "start_stamp": "2023-11-25T05:05:45.367937+0000", 2023-11-25T05:06:06.671 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "gid": 4545, 2023-11-25T05:06:06.671 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "addr": "172.21.15.7:0/2475409171", 2023-11-25T05:06:06.671 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "metadata": { 2023-11-25T05:06:06.671 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "arch": "x86_64", 2023-11-25T05:06:06.671 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "ceph_release": "reef", 2023-11-25T05:06:06.671 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "ceph_version": "ceph version 18.0.0-7506-gec928368 (ec928368edcc5fc71884a56fd9fd821b148f4e30) reef (dev)", 2023-11-25T05:06:06.671 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "ceph_version_short": "18.0.0-7506-gec928368", 2023-11-25T05:06:06.671 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2023-11-25T05:06:08.141 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:06:08.133+0000 10252640 -1 osd.4 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4537.0:2 22.1a 22:5c6776db:::0.obj:head [write 0~202 in=202b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-11-25T05:06:09.513 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:06:09.473+0000 10252640 -1 osd.4 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4537.0:2 22.1a 22:5c6776db:::0.obj:head [write 0~202 in=202b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-11-25T05:06:09.690 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:06:09.654+0000 10a53640 -1 osd.2 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4441.0:2 7.1 7:81573e0d:test-rados-api-smithi007-92746-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-25T05:06:10.077 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-25T05:06:10.057+0000 f250640 -1 osd.0 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4501.0:2 11.1 11:81bcbe23:test-rados-api-smithi007-93021-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-25T05:06:10.890 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:06:10.858+0000 10a53640 -1 osd.2 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4441.0:2 7.1 7:81573e0d:test-rados-api-smithi007-92746-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-25T05:06:11.360 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:06:11.270+0000 10252640 -1 osd.4 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4537.0:2 22.1a 22:5c6776db:::0.obj:head [write 0~202 in=202b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-11-25T05:06:11.629 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-25T05:06:11.620+0000 f250640 -1 osd.0 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4501.0:2 11.1 11:81bcbe23:test-rados-api-smithi007-93021-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-25T05:06:12.017 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:06:11.945+0000 10a53640 -1 osd.2 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4441.0:2 7.1 7:81573e0d:test-rados-api-smithi007-92746-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-25T05:06:12.351 INFO:tasks.workunit.client.0.smithi007.stdout: a api_misc: [==========] Running 12 tests from 4 test suites. 2023-11-25T05:06:12.351 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] Global test environment set-up. 2023-11-25T05:06:12.351 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2023-11-25T05:06:12.351 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2023-11-25T05:06:12.351 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2023-11-25T05:06:12.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-11-25T05:06:12.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2023-11-25T05:06:12.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2023-11-25T05:06:12.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2023-11-25T05:06:12.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2023-11-25T05:06:12.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2023-11-25T05:05:36.079+0000 7fd06c986780 -1 failed for service _ceph-mon._tcp 2023-11-25T05:06:12.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2023-11-25T05:05:36.079+0000 7fd06c986780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2023-11-25T05:06:12.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (452 ms) 2023-11-25T05:06:12.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2023-11-25T05:06:12.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2009 ms) 2023-11-25T05:06:12.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2461 ms total) 2023-11-25T05:06:12.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2023-11-25T05:06:12.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2023-11-25T05:06:12.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2023-11-25T05:06:12.352 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8476171e0 2023-11-25T05:06:12.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d846bacd30 2023-11-25T05:06:12.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d847616140 2023-11-25T05:06:12.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8476b1a50 2023-11-25T05:06:12.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8475fa920 2023-11-25T05:06:12.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8469cbd40 2023-11-25T05:06:12.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8469c7ca0 2023-11-25T05:06:12.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8476be930 2023-11-25T05:06:12.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8476bf2e0 2023-11-25T05:06:12.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8476bfd00 2023-11-25T05:06:12.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8476bf120 2023-11-25T05:06:12.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8476c0c50 2023-11-25T05:06:12.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8476bfac0 2023-11-25T05:06:12.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8476c1af0 2023-11-25T05:06:12.353 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8476c09a0 2023-11-25T05:06:12.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8476c17f0 2023-11-25T05:06:12.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d846d25a00 2023-11-25T05:06:12.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d846d26420 2023-11-25T05:06:12.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8476c2770 2023-11-25T05:06:12.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d846d26fb0 2023-11-25T05:06:12.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d846d27a20 2023-11-25T05:06:12.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d846d261a0 2023-11-25T05:06:12.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d846d28870 2023-11-25T05:06:12.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d846d277f0 2023-11-25T05:06:12.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d846d29780 2023-11-25T05:06:12.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d846d28640 2023-11-25T05:06:12.354 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d846d2a600 2023-11-25T05:06:12.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d846d29490 2023-11-25T05:06:12.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8475fdd00 2023-11-25T05:06:12.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d8475fe7c0 2023-11-25T05:06:12.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x55d846d2a3f0 2023-11-25T05:06:12.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 31 aios 2023-11-25T05:06:12.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8476171e0 2023-11-25T05:06:12.355 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d846bacd30 2023-11-25T05:06:12.356 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d847616140 2023-11-25T05:06:12.359 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8476b1a50 2023-11-25T05:06:12.360 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8475fa920 2023-11-25T05:06:12.360 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8469cbd40 2023-11-25T05:06:12.360 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8469c7ca0 2023-11-25T05:06:12.360 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8476be930 2023-11-25T05:06:12.360 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8476bf2e0 2023-11-25T05:06:12.360 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8476bfd00 2023-11-25T05:06:12.361 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8476bf120 2023-11-25T05:06:12.361 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8476c0c50 2023-11-25T05:06:12.361 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8476bfac0 2023-11-25T05:06:12.361 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8476c1af0 2023-11-25T05:06:12.361 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8476c09a0 2023-11-25T05:06:12.361 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8476c17f0 2023-11-25T05:06:12.361 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d846d25a00 2023-11-25T05:06:12.362 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d846d26420 2023-11-25T05:06:12.362 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8476c2770 2023-11-25T05:06:12.688 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:06:12.677+0000 10252640 -1 osd.4 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4537.0:2 22.1a 22:5c6776db:::0.obj:head [write 0~202 in=202b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-11-25T05:06:13.011 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-25T05:06:13.007+0000 f250640 -1 osd.0 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4501.0:2 11.1 11:81bcbe23:test-rados-api-smithi007-93021-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-25T05:06:13.657 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:06:13.482+0000 10a53640 -1 osd.2 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4441.0:2 7.1 7:81573e0d:test-rados-api-smithi007-92746-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-25T05:06:14.008 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:06:13.927+0000 10252640 -1 osd.4 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4537.0:2 22.1a 22:5c6776db:::0.obj:head [write 0~202 in=202b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-11-25T05:06:14.157 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:56.355 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:14.229 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:56.425 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:14.230 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:56.427 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:14.245 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:56.443 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:14.251 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:56.394 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:14.265 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:56.409 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:14.384 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:56.582 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:14.443 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-25T05:06:14.350+0000 f250640 -1 osd.0 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4501.0:2 11.1 11:81bcbe23:test-rados-api-smithi007-93021-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-25T05:06:14.448 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:56.592 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:14.504 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:56.703 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:14.539 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:06:14.504+0000 10a53640 -1 osd.2 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4441.0:2 7.1 7:81573e0d:test-rados-api-smithi007-92746-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-25T05:06:14.974 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:06:14.957+0000 10252640 -1 osd.4 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4537.0:2 22.1a 22:5c6776db:::0.obj:head [write 0~202 in=202b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-11-25T05:06:15.452 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-25T05:06:15.357+0000 f250640 -1 osd.0 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4501.0:2 11.1 11:81bcbe23:test-rados-api-smithi007-93021-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-25T05:06:16.083 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:06:16.062+0000 10252640 -1 osd.4 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4537.0:2 22.1a 22:5c6776db:::0.obj:head [write 0~202 in=202b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-11-25T05:06:17.264 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55pi_service_pp: "distro": "centos", 2023-11-25T05:06:17.264 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-11-25T05:06:17.264 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "distro_version": "9", 2023-11-25T05:06:17.264 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "foo": "bar", 2023-11-25T05:06:17.264 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "hostname": "smithi007", 2023-11-25T05:06:17.264 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Nov 15 10:38:22 UTC 2023", 2023-11-25T05:06:17.264 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "kernel_version": "5.14.0-386.el9.x86_64", 2023-11-25T05:06:17.264 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "mem_swap_kb": "0", 2023-11-25T05:06:17.264 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "mem_total_kb": "32494892", 2023-11-25T05:06:17.265 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "os": "Linux", 2023-11-25T05:06:17.265 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "this": "that" 2023-11-25T05:06:17.265 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: }, 2023-11-25T05:06:17.265 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "task_status": {} 2023-11-25T05:06:17.265 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: } 2023-11-25T05:06:17.265 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: } 2023-11-25T05:06:17.265 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: } 2023-11-25T05:06:17.265 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: } 2023-11-25T05:06:17.265 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: } 2023-11-25T05:06:17.265 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: 2023-11-25T05:06:17.265 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: attempt 1 of 20 2023-11-25T05:06:17.265 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: failed to deregister: 2023-11-25T05:06:17.265 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: { 2023-11-25T05:06:17.266 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "epoch": 14, 2023-11-25T05:06:17.266 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "modified": "2023-11-25T05:06:07.400850+0000", 2023-11-25T05:06:17.266 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "services": { 2023-11-25T05:06:17.266 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "laundry": { 2023-11-25T05:06:17.266 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "daemons": { 2023-11-25T05:06:17.266 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "summary": "", 2023-11-25T05:06:17.266 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "close-test-pid93323": { 2023-11-25T05:06:17.266 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "start_epoch": 14, 2023-11-25T05:06:17.266 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "start_stamp": "2023-11-25T05:06:06.950290+0000", 2023-11-25T05:06:17.266 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "gid": 4556, 2023-11-25T05:06:17.266 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "addr": "172.21.15.7:0/573557720", 2023-11-25T05:06:17.266 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "metadata": { 2023-11-25T05:06:17.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "arch": "x86_64", 2023-11-25T05:06:17.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "ceph_release": "reef", 2023-11-25T05:06:17.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "ceph_version": "ceph version 18.0.0-7506-gec928368 (ec928368edcc5fc71884a56fd9fd821b148f4e30) reef (dev)", 2023-11-25T05:06:17.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "ceph_version_short": "18.0.0-7506-gec928368", 2023-11-25T05:06:17.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2023-11-25T05:06:17.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "distro": "centos", 2023-11-25T05:06:17.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-11-25T05:06:17.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "distro_version": "9", 2023-11-25T05:06:17.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "foo": "bar", 2023-11-25T05:06:17.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "hostname": "smithi007", 2023-11-25T05:06:17.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Nov 15 10:38:22 UTC 2023", 2023-11-25T05:06:17.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "kernel_version": "5.14.0-386.el9.x86_64", 2023-11-25T05:06:17.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "mem_swap_kb": "0", 2023-11-25T05:06:17.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "mem_total_kb": "32494892", 2023-11-25T05:06:17.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "os": "Linux", 2023-11-25T05:06:17.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "this": "that" 2023-11-25T05:06:17.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: }, 2023-11-25T05:06:17.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "task_status": {} 2023-11-25T05:06:17.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: } 2023-11-25T05:06:17.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: } 2023-11-25T05:06:17.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: } 2023-11-25T05:06:17.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: } 2023-11-25T05:06:17.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: } 2023-11-25T05:06:17.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: 2023-11-25T05:06:17.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: attempt 2 of 20 2023-11-25T05:06:17.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (15870 ms) 2023-11-25T05:06:17.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (41419 ms total) 2023-11-25T05:06:17.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: 2023-11-25T05:06:17.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [----------] Global test environment tear-down 2023-11-25T05:06:17.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (41419 ms total) 2023-11-25T05:06:17.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ PASSED ] 4 tests. 2023-11-25T05:06:17.620 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:06:17.612+0000 10252640 -1 osd.4 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4537.0:2 22.1a 22:5c6776db:::0.obj:head [write 0~202 in=202b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-11-25T05:06:17.897 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-25T05:06:17.784+0000 10252640 -1 osd.1 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:1 27.1s0 27:848861ac:test-rados-api-smithi007-93181-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-11-25T05:06:18.770 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:06:18.676+0000 10252640 -1 osd.4 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4537.0:2 22.1a 22:5c6776db:::0.obj:head [write 0~202 in=202b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-11-25T05:06:18.882 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-25T05:06:18.870+0000 10252640 -1 osd.1 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:1 27.1s0 27:848861ac:test-rados-api-smithi007-93181-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-11-25T05:06:19.336 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:06:19.253+0000 10a53640 -1 osd.2 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4494.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-11-25T05:06:19.754 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:06:19.731+0000 10252640 -1 osd.4 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4537.0:2 22.1a 22:5c6776db:::0.obj:head [write 0~202 in=202b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-11-25T05:06:21.003 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:06:20.955+0000 10a53640 -1 osd.2 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4494.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-11-25T05:06:22.092 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:06:22.082+0000 10a53640 -1 osd.2 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4494.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-11-25T05:06:23.240 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:06:23.234+0000 10a53640 -1 osd.2 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4494.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-11-25T05:06:24.816 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:06:24.786+0000 10a53640 -1 osd.2 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4494.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-11-25T05:06:25.835 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: Running main() from gmock_main.cc 2023-11-25T05:06:25.835 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2023-11-25T05:06:25.835 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [----------] Global test environment set-up. 2023-11-25T05:06:25.835 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [----------] 9 tests from LibRadosPools 2023-11-25T05:06:25.835 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2023-11-25T05:06:25.835 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolList (4311 ms) 2023-11-25T05:06:25.835 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2023-11-25T05:06:25.835 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3995 ms) 2023-11-25T05:06:25.835 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2023-11-25T05:06:25.835 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3790 ms) 2023-11-25T05:06:25.836 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2023-11-25T05:06:25.836 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (4768 ms) 2023-11-25T05:06:25.836 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2023-11-25T05:06:25.836 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (5004 ms) 2023-11-25T05:06:25.836 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2023-11-25T05:06:25.836 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6168 ms) 2023-11-25T05:06:25.836 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2023-11-25T05:06:25.836 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (6952 ms) 2023-11-25T05:06:25.836 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2023-11-25T05:06:25.836 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5789 ms) 2023-11-25T05:06:25.836 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2023-11-25T05:06:25.836 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (9397 ms) 2023-11-25T05:06:25.837 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [----------] 9 tests from LibRadosPools (50175 ms total) 2023-11-25T05:06:25.837 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: 2023-11-25T05:06:25.837 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [----------] Global test environment tear-down 2023-11-25T05:06:25.837 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (50175 ms total) 2023-11-25T05:06:25.837 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ PASSED ] 9 tests. 2023-11-25T05:06:27.445 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:06:27.339+0000 10a53640 -1 osd.2 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4494.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27) 2023-11-25T05:06:28.118 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-25T05:06:28.103+0000 10252640 -1 osd.1 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:1 27.1s0 27:848861ac:test-rados-api-smithi007-93181-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-11-25T05:06:29.157 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-25T05:06:29.137+0000 10252640 -1 osd.1 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:1 27.1s0 27:848861ac:test-rados-api-smithi007-93181-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-11-25T05:06:39.424 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:21.622 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:39.428 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:03:21.572 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:39.436 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:21.634 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:39.442 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:21.640 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:39.484 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:21.682 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:39.491 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:03:21.635 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:39.657 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:03:21.801 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:39.703 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:21.901 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:39.846 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:22.044 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:06:44.108 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: Running main() from gmock_main.cc 2023-11-25T05:06:46.699 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2023-11-25T05:06:46.700 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] Global test environment set-up. 2023-11-25T05:06:46.700 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] 5 tests from LibRadosStat 2023-11-25T05:06:46.700 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2023-11-25T05:06:46.700 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStat.Stat (19668 ms) 2023-11-25T05:06:46.700 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2023-11-25T05:06:46.700 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (3723 ms) 2023-11-25T05:06:46.700 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2023-11-25T05:06:46.700 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStat.StatNS (5732 ms) 2023-11-25T05:06:46.700 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2023-11-25T05:06:46.700 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (195 ms) 2023-11-25T05:06:46.700 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2023-11-25T05:06:46.700 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (3345 ms) 2023-11-25T05:06:46.700 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] 5 tests from LibRadosStat (32664 ms total) 2023-11-25T05:06:46.700 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: 2023-11-25T05:06:46.701 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2023-11-25T05:06:46.701 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2023-11-25T05:06:46.701 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (16198 ms) 2023-11-25T05:06:46.701 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2023-11-25T05:06:46.701 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (7131 ms) 2023-11-25T05:06:46.701 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2023-11-25T05:06:46.701 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (14 ms) 2023-11-25T05:06:46.701 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2023-11-25T05:06:46.701 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1857 ms) 2023-11-25T05:06:46.701 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (25200 ms total) 2023-11-25T05:06:46.701 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: 2023-11-25T05:06:46.701 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] Global test environment tear-down 2023-11-25T05:06:46.701 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (68384 ms total) 2023-11-25T05:06:46.701 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ PASSED ] 9 tests. 2023-11-25T05:06:46.702 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2023-11-25T05:06:46.702 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2023-11-25T05:06:46.702 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2023-11-25T05:06:46.702 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2023-11-25T05:06:46.702 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (69751 ms) 2023-11-25T05:06:46.703 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (69751 ms total) 2023-11-25T05:06:46.703 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: 2023-11-25T05:06:46.703 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2023-11-25T05:06:46.703 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (69751 ms total) 2023-11-25T05:06:46.703 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2023-11-25T05:07:01.809 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: Running main() from gmock_main.cc 2023-11-25T05:07:01.809 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2023-11-25T05:07:01.809 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [----------] Global test environment set-up. 2023-11-25T05:07:01.809 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2023-11-25T05:07:01.809 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2023-11-25T05:07:01.809 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (736 ms) 2023-11-25T05:07:01.809 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2023-11-25T05:07:01.809 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (86 ms) 2023-11-25T05:07:01.809 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2023-11-25T05:07:01.810 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (85175 ms) 2023-11-25T05:07:01.810 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (85997 ms total) 2023-11-25T05:07:01.810 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: 2023-11-25T05:07:01.810 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [----------] Global test environment tear-down 2023-11-25T05:07:01.810 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (85997 ms total) 2023-11-25T05:07:01.810 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2023-11-25T05:07:05.728 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: Running main() from gmock_main.cc 2023-11-25T05:07:05.728 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2023-11-25T05:07:05.728 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [----------] Global test environment set-up. 2023-11-25T05:07:05.728 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2023-11-25T05:07:05.728 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2023-11-25T05:07:05.728 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2023-11-25T05:07:05.728 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2023-11-25T05:07:05.729 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (12687 ms) 2023-11-25T05:07:05.729 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2023-11-25T05:07:05.729 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (15919 ms) 2023-11-25T05:07:05.729 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2023-11-25T05:07:05.729 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (3945 ms) 2023-11-25T05:07:05.729 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2023-11-25T05:07:05.729 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (29364 ms) 2023-11-25T05:07:05.729 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2023-11-25T05:07:05.729 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (10202 ms) 2023-11-25T05:07:05.729 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2023-11-25T05:07:05.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (1267 ms) 2023-11-25T05:07:05.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2023-11-25T05:07:05.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (915 ms) 2023-11-25T05:07:05.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2023-11-25T05:07:05.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (1784 ms) 2023-11-25T05:07:05.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2023-11-25T05:07:05.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (147 ms) 2023-11-25T05:07:05.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2023-11-25T05:07:05.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (215 ms) 2023-11-25T05:07:05.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2023-11-25T05:07:05.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (236 ms) 2023-11-25T05:07:05.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2023-11-25T05:07:05.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (204 ms) 2023-11-25T05:07:05.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2023-11-25T05:07:05.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (683 ms) 2023-11-25T05:07:05.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2023-11-25T05:07:05.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (588 ms) 2023-11-25T05:07:05.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2023-11-25T05:07:05.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (300 ms) 2023-11-25T05:07:05.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2023-11-25T05:07:05.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (115 ms) 2023-11-25T05:07:05.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (78571 ms total) 2023-11-25T05:07:05.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: 2023-11-25T05:07:05.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [----------] Global test environment tear-down 2023-11-25T05:07:05.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (89889 ms total) 2023-11-25T05:07:05.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2023-11-25T05:07:12.644 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: Running main() from gmock_main.cc 2023-11-25T05:07:12.644 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2023-11-25T05:07:12.644 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [----------] Global test environment set-up. 2023-11-25T05:07:12.644 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2023-11-25T05:07:12.644 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2023-11-25T05:07:12.644 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (1253 ms) 2023-11-25T05:07:12.644 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2023-11-25T05:07:12.644 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (82 ms) 2023-11-25T05:07:12.644 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2023-11-25T05:07:12.644 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (84937 ms) 2023-11-25T05:07:12.645 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2023-11-25T05:07:12.645 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:06:53.376648+0000 mon.c [INF] from='client.? 172.21.15.7:0/2732281325' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi007-93181-6", "pg_num": 4}]: dispatch 2023-11-25T05:07:12.645 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:06:54.696040+0000 mon.c [INF] from='client.? 172.21.15.7:0/2246502139' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi007-92908-6"}]: dispatch 2023-11-25T05:07:12.645 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:06:58.252603+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi007-93181-6", "pg_num": 4}]: dispatch 2023-11-25T05:07:12.645 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:06:58.351858+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi007-92908-6"}]: dispatch 2023-11-25T05:07:12.645 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:01.284967+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi007-93181-6", "pg_num": 4}]': finished 2023-11-25T05:07:12.645 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:01.286777+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi007-92908-6"}]': finished 2023-11-25T05:07:12.645 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:02.717055+0000 mon.c [INF] from='client.? 172.21.15.7:0/2732281325' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi007-93181-4", "tierpool": "test-rados-api-smithi007-93181-6", "force_nonempty": "--force-nonempty" }]: dispatch 2023-11-25T05:07:12.645 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:02.811667+0000 mon.c [INF] from='client.? 172.21.15.7:0/2246502139' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi007-92908-6", "tierpool": "test-rados-api-smithi007-92908-7"}]: dispatch 2023-11-25T05:07:12.645 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:04.124567+0000 mon.b [INF] from='client.? 172.21.15.7:0/2895637444' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi007-92898-10"}]: dispatch 2023-11-25T05:07:12.645 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:04.167978+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi007-93181-4", "tierpool": "test-rados-api-smithi007-93181-6", "force_nonempty": "--force-nonempty" }]: dispatch 2023-11-25T05:07:12.645 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:04.186556+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi007-92908-6", "tierpool": "test-rados-api-smithi007-92908-7"}]: dispatch 2023-11-25T05:07:12.646 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:05.475582+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi007-92898-10"}]: dispatch 2023-11-25T05:07:12.646 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:05.842723+0000 mon.a [INF] from='client.? 172.21.15.7:0/2716537066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi007-93021-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-11-25T05:07:13.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11- api_list: [==========] Running 11 tests from 3 test suites. 2023-11-25T05:07:13.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] Global test environment set-up. 2023-11-25T05:07:13.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] 7 tests from LibRadosList 2023-11-25T05:07:13.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2023-11-25T05:07:13.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.ListObjects (75951 ms) 2023-11-25T05:07:13.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2023-11-25T05:07:13.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (1147 ms) 2023-11-25T05:07:13.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2023-11-25T05:07:13.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset foo1,foo2,foo3 2023-11-25T05:07:13.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo1 2023-11-25T05:07:13.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo2 2023-11-25T05:07:13.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo3 2023-11-25T05:07:13.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset foo1,foo4,foo5 2023-11-25T05:07:13.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo4 2023-11-25T05:07:13.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo5 2023-11-25T05:07:13.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo1 2023-11-25T05:07:13.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset foo6,foo7 2023-11-25T05:07:13.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo7 2023-11-25T05:07:13.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo6 2023-11-25T05:07:13.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-11-25T05:07:13.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns1:foo4 2023-11-25T05:07:13.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns1:foo5 2023-11-25T05:07:13.319 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns2:foo7 2023-11-25T05:07:13.319 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns2:foo6 2023-11-25T05:07:13.319 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns1:foo1 2023-11-25T05:07:13.319 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: :foo1 2023-11-25T05:07:13.319 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: :foo2 2023-11-25T05:07:13.319 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: :foo3 2023-11-25T05:07:13.319 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (2581 ms) 2023-11-25T05:07:13.319 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2023-11-25T05:07:13.319 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 1 0 2023-11-25T05:07:13.319 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 10 0 2023-11-25T05:07:13.319 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 13 0 2023-11-25T05:07:13.319 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 7 0 2023-11-25T05:07:13.319 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 14 0 2023-11-25T05:07:13.319 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 0 0 2023-11-25T05:07:13.320 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 15 0 2023-11-25T05:07:13.320 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 11 0 2023-11-25T05:07:13.320 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 5 0 2023-11-25T05:07:13.320 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 8 0 2023-11-25T05:07:13.320 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 6 0 2023-11-25T05:07:13.320 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 3 0 2023-11-25T05:07:13.320 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 4 0 2023-11-25T05:07:13.320 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 12 0 2023-11-25T05:07:13.320 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 9 0 2023-11-25T05:07:13.320 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 2 0 2023-11-25T05:07:13.320 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-11-25T05:07:13.320 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (10793 ms) 2023-11-25T05:07:13.320 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2023-11-25T05:07:13.320 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: x cursor=MIN 2023-11-25T05:07:13.320 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=1 cursor=13:02547ec2:::1:head 2023-11-25T05:07:13.321 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=10 cursor=13:52ea6a34:::10:head 2023-11-25T05:07:13.321 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=13 cursor=13:566253c9:::13:head 2023-11-25T05:07:13.321 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=7 cursor=13:5c6b0b28:::7:head 2023-11-25T05:07:13.321 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=14 cursor=13:62a1935d:::14:head 2023-11-25T05:07:13.321 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=0 cursor=13:6cac518f:::0:head 2023-11-25T05:07:13.321 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=15 cursor=13:863748b0:::15:head 2023-11-25T05:07:13.321 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=11 cursor=13:89d3ae78:::11:head 2023-11-25T05:07:13.321 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=5 cursor=13:b29083e3:::5:head 2023-11-25T05:07:13.321 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=8 cursor=13:bd63b0f1:::8:head 2023-11-25T05:07:13.321 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=6 cursor=13:c4fdafeb:::6:head 2023-11-25T05:07:13.321 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=3 cursor=13:cfc208b3:::3:head 2023-11-25T05:07:13.322 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=4 cursor=13:d83876eb:::4:head 2023-11-25T05:07:13.322 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=12 cursor=13:de5d7c5f:::12:head 2023-11-25T05:07:13.322 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=9 cursor=13:e960b815:::9:head 2023-11-25T05:07:13.322 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=2 cursor=13:f905c69b:::2:head 2023-11-25T05:07:13.322 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: FIRST> seek to MIN oid=1 2023-11-25T05:07:13.322 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=1 cursor=13:02547ec2:::1:head 2023-11-25T05:07:13.322 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:02547ec2:::1:head 2023-11-25T05:07:13.322 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:02547ec2:::1:head -> 1 2023-11-25T05:07:13.322 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=10 cursor=13:52ea6a34:::10:head 2023-11-25T05:07:13.322 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:52ea6a34:::10:head 2023-11-25T05:07:13.322 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2023-11-25T05:07:13.323 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=13 cursor=13:566253c9:::13:head 2023-11-25T05:07:13.323 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:566253c9:::13:head 2023-11-25T05:07:13.323 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:566253c9:::13:head -> 13 2023-11-25T05:07:13.323 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=7 cursor=13:5c6b0b28:::7:head 2023-11-25T05:07:13.323 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2023-11-25T05:07:13.323 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2023-11-25T05:07:17.399 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=14 cursor=13:62a1935d:::14:head 2023-11-25T05:07:17.399 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:62a1935d:::14:head 2023-11-25T05:07:17.399 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:62a1935d:::14:head -> 14 2023-11-25T05:07:17.399 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=0 cursor=13:6cac518f:::0:head 2023-11-25T05:07:17.399 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:6cac518f:::0:head 2023-11-25T05:07:17.399 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:6cac518f:::0:head -> 0 2023-11-25T05:07:17.399 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=15 cursor=13:863748b0:::15:head 2023-11-25T05:07:17.399 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:863748b0:::15:head 2023-11-25T05:07:17.399 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:863748b0:::15:head -> 15 2023-11-25T05:07:17.399 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=11 cursor=13:89d3ae78:::11:head 2023-11-25T05:07:17.400 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:89d3ae78:::11:head 2023-11-25T05:07:17.400 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2023-11-25T05:07:17.400 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=5 cursor=13:b29083e3:::5:head 2023-11-25T05:07:17.400 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:b29083e3:::5:head 2023-11-25T05:07:17.400 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:b29083e3:::5:head -> 5 2023-11-25T05:07:17.400 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=8 cursor=13:bd63b0f1:::8:head 2023-11-25T05:07:17.400 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2023-11-25T05:07:17.400 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2023-11-25T05:07:17.400 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=6 cursor=13:c4fdafeb:::6:head 2023-11-25T05:07:17.400 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2023-11-25T05:07:17.400 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2023-11-25T05:07:17.400 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=3 cursor=13:cfc208b3:::3:head 2023-11-25T05:07:17.400 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:cfc208b3:::3:head 2023-11-25T05:07:17.401 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2023-11-25T05:07:17.401 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=4 cursor=13:d83876eb:::4:head 2023-11-25T05:07:17.401 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:d83876eb:::4:head 2023-11-25T05:07:17.401 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:d83876eb:::4:head -> 4 2023-11-25T05:07:17.401 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=12 cursor=13:de5d7c5f:::12:head 2023-11-25T05:07:17.401 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2023-11-25T05:07:17.401 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2023-11-25T05:07:17.401 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=9 cursor=13:e960b815:::9:head 2023-11-25T05:07:17.401 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:e960b815:::9:head 2023-11-25T05:07:17.401 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:e960b815:::9:head -> 9 2023-11-25T05:07:17.401 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=2 cursor=13:f905c69b:::2:head 2023-11-25T05:07:17.401 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:f905c69b:::2:head 2023-11-25T05:07:17.401 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:f905c69b:::2:head -> 2 2023-11-25T05:07:17.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2023-11-25T05:07:17.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:de5d7c5f:::12:head expected=13:de5d7c5f:::12:head 2023-11-25T05:07:17.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2023-11-25T05:07:17.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=12 expected=12 2023-11-25T05:07:17.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:d83876eb:::4:head 2023-11-25T05:07:17.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:d83876eb:::4:head expected=13:d83876eb:::4:head 2023-11-25T05:07:17.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:d83876eb:::4:head -> 4 2023-11-25T05:07:17.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=4 expected=4 2023-11-25T05:07:17.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:cfc208b3:::3:head 2023-11-25T05:07:17.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:cfc208b3:::3:head expected=13:cfc208b3:::3:head 2023-11-25T05:07:17.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2023-11-25T05:07:17.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=3 expected=3 2023-11-25T05:07:17.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2023-11-25T05:07:17.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:c4fdafeb:::6:head expected=13:c4fdafeb:::6:head 2023-11-25T05:07:17.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2023-11-25T05:07:17.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=6 expected=6 2023-11-25T05:07:17.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2023-11-25T05:07:17.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:5c6b0b28:::7:head expected=13:5c6b0b28:::7:head 2023-11-25T05:07:17.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2023-11-25T05:07:17.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=7 expected=7 2023-11-25T05:07:17.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:566253c9:::13:head 2023-11-25T05:07:17.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:566253c9:::13:head expected=13:566253c9:::13:head 2023-11-25T05:07:17.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:566253c9:::13:head -> 13 2023-11-25T05:07:17.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=13 expected=13 2023-11-25T05:07:17.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:52ea6a34:::10:head 2023-11-25T05:07:17.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:52ea6a34:::10:head expected=13:52ea6a34:::10:head 2023-11-25T05:07:17.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2023-11-25T05:07:17.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=10 expected=10 2023-11-25T05:07:17.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:02547ec2:::1:head 2023-11-25T05:07:17.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:02547ec2:::1:head expected=13:02547ec2:::1:head 2023-11-25T05:07:17.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:02547ec2:::1:head -> 1 2023-11-25T05:07:21.772 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:03.970 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:21.794 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:03.992 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:22.184 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:04.382 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:22.506 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:04.704 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:22.523 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:04.717 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:22.548 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:04:04.692 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:22.563 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:04:04.707 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:22.756 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:04:04.900 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:22.759 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:04.956 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:23.230 INFO:tasks.workunit.client.0.smithi007.stdout: api_25T05:07:04.956133+0000 mon.b [INF] from='client.? 172.21.15.7:0/2632810160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi007-92746-22"}]: dispatch 2023-11-25T05:07:23.231 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:05.305970+0000 mon.c [INF] from='client.? 172.21.15.7:0/2704597848' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-11-25T05:07:23.231 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:05.931247+0000 mon.b [INF] from='client.? 172.21.15.7:0/2095359967' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi007-92726-16"}]: dispatch 2023-11-25T05:07:23.231 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:06.522759+0000 mon.a [WRN] Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-11-25T05:07:23.231 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:06.759475+0000 client.admin [INF] onexx 2023-11-25T05:07:23.231 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:06.770971+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-11-25T05:07:23.231 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:06.982128+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi007-92746-22"}]: dispatch 2023-11-25T05:07:23.231 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:06.997081+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi007-92726-16"}]: dispatch 2023-11-25T05:07:23.231 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:07.009019+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi007-93181-4", "tierpool": "test-rados-api-smithi007-93181-6", "force_nonempty": "--force-nonempty" }]': finished 2023-11-25T05:07:23.231 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:07.012207+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi007-92908-6", "tierpool": "test-rados-api-smithi007-92908-7"}]': finished 2023-11-25T05:07:23.231 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:08.182571+0000 mon.b [INF] from='client.? 172.21.15.7:0/2095359967' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi007-92726-16"}]: dispatch 2023-11-25T05:07:23.231 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:08.332597+0000 mon.b [INF] from='client.? 172.21.15.7:0/2632810160' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi007-92746-22"}]: dispatch 2023-11-25T05:07:23.231 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:08.420571+0000 mon.b [INF] from='client.? 172.21.15.7:0/2895637444' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi007-92898-10"}]: dispatch 2023-11-25T05:07:23.232 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:15.024844+0000 mon.b [INF] from='client.? 172.21.15.7:0/885257416' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi007-93106-7"}]: dispatch 2023-11-25T05:07:23.232 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:17.061247+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi007-93106-7"}]: dispatch 2023-11-25T05:07:23.232 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:17.156917+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi007-93181-4", "overlaypool": "test-rados-api-smithi007-93181-6"}]': finished 2023-11-25T05:07:23.232 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:17.162770+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi007-92898-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-11-25T05:07:24.412 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:17.166031+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi007-92746-22", "profile": [ "k=2", "m=1", "crush-f api_stat_pp: Running main() from gmock_main.cc 2023-11-25T05:07:24.412 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2023-11-25T05:07:24.412 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] Global test environment set-up. 2023-11-25T05:07:24.412 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2023-11-25T05:07:24.412 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: seed 93106 2023-11-25T05:07:24.413 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2023-11-25T05:07:24.413 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (34574 ms) 2023-11-25T05:07:24.413 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2023-11-25T05:07:24.413 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (7855 ms) 2023-11-25T05:07:24.413 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2023-11-25T05:07:24.413 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (196 ms) 2023-11-25T05:07:24.413 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2023-11-25T05:07:24.413 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (6895 ms) 2023-11-25T05:07:24.413 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2023-11-25T05:07:24.413 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (13057 ms) 2023-11-25T05:07:24.413 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (62577 ms total) 2023-11-25T05:07:24.414 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: 2023-11-25T05:07:24.414 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2023-11-25T05:07:24.414 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2023-11-25T05:07:24.414 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (8321 ms) 2023-11-25T05:07:24.414 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2023-11-25T05:07:24.414 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (4760 ms) 2023-11-25T05:07:24.414 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2023-11-25T05:07:24.414 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (1599 ms) 2023-11-25T05:07:24.414 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2023-11-25T05:07:24.414 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (2358 ms) 2023-11-25T05:07:24.414 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (17038 ms total) 2023-11-25T05:07:24.415 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: 2023-11-25T05:07:24.415 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] Global test environment tear-down 2023-11-25T05:07:24.415 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (108646 ms total) 2023-11-25T05:07:24.415 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ PASSED ] 9 tests. 2023-11-25T05:07:26.047 INFO:tasks.workunit.client.0.smithi007.stdout:ailure-domain=osd"]}]': finished 2023-11-25T05:07:26.047 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:17.169040+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi007-92726-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-11-25T05:07:26.047 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:17.172187+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi007-93350-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-11-25T05:07:26.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:17.174926+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi007-92688-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-11-25T05:07:26.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:17.454568+0000 mon.c [INF] from='client.? 172.21.15.7:0/1375230274' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi007-93164-12"}]: dispatch 2023-11-25T05:07:26.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:18.890845+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-11-25T05:07:26.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:19.187996+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi007-93164-12"}]: dispatch 2023-11-25T05:07:26.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:19.611983+0000 mon.a [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-11-25T05:07:26.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:18.388223+0000 mon.b [INF] from='client.? 172.21.15.7:0/2895637444' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi007-92898-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi007-92898-10"}]: dispatch 2023-11-25T05:07:26.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:18.528709+0000 mon.b [INF] from='client.? 172.21.15.7:0/2632810160' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi007-92746-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi007-92746-22"}]: dispatch 2023-11-25T05:07:26.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:18.631534+0000 mon.c [INF] from='client.? 172.21.15.7:0/2732281325' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi007-93181-6", "mode": "writeback"}]: dispatch 2023-11-25T05:07:26.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:18.757593+0000 mon.b [INF] from='client.? 172.21.15.7:0/2095359967' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi007-92726-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi007-92726-16"}]: dispatch 2023-11-25T05:07:26.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:19.141134+0000 mon.b [INF] from='client.? 172.21.15.7:0/2530874621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi007-92842-10"}]: dispatch 2023-11-25T05:07:26.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:19.998403+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi007-92898-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi007-92898-10"}]: dispatch 2023-11-25T05:07:26.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:20.039228+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi007-92746-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi007-92746-22"}]: dispatch 2023-11-25T05:07:32.989 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:20.051082+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi007-92726-16", "pool_type":"er api_lock: Running main() from gmock_main.cc 2023-11-25T05:07:32.989 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2023-11-25T05:07:32.989 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] Global test environment set-up. 2023-11-25T05:07:32.989 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] 8 tests from LibRadosLock 2023-11-25T05:07:32.989 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2023-11-25T05:07:32.989 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (13548 ms) 2023-11-25T05:07:32.989 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2023-11-25T05:07:32.989 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.LockShared (2192 ms) 2023-11-25T05:07:32.990 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2023-11-25T05:07:32.990 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (7408 ms) 2023-11-25T05:07:32.990 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2023-11-25T05:07:32.990 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (4360 ms) 2023-11-25T05:07:32.990 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2023-11-25T05:07:32.990 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (3689 ms) 2023-11-25T05:07:32.990 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2023-11-25T05:07:32.990 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.Unlock (3092 ms) 2023-11-25T05:07:32.990 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2023-11-25T05:07:32.990 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (19779 ms) 2023-11-25T05:07:32.990 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2023-11-25T05:07:32.990 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (9423 ms) 2023-11-25T05:07:32.990 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] 8 tests from LibRadosLock (63491 ms total) 2023-11-25T05:07:32.991 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: 2023-11-25T05:07:32.991 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2023-11-25T05:07:32.991 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2023-11-25T05:07:32.991 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (8445 ms) 2023-11-25T05:07:32.991 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2023-11-25T05:07:32.991 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (427 ms) 2023-11-25T05:07:32.991 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2023-11-25T05:07:32.991 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (9523 ms) 2023-11-25T05:07:32.991 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2023-11-25T05:07:32.991 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (2580 ms) 2023-11-25T05:07:32.991 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2023-11-25T05:07:32.991 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (607 ms) 2023-11-25T05:07:32.991 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2023-11-25T05:07:32.991 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (289 ms) 2023-11-25T05:07:32.992 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2023-11-25T05:07:32.992 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (1175 ms) 2023-11-25T05:07:32.992 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2023-11-25T05:07:32.992 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (137 ms) 2023-11-25T05:07:32.992 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (23183 ms total) 2023-11-25T05:07:32.992 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: 2023-11-25T05:07:32.992 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] Global test environment tear-down 2023-11-25T05:07:32.992 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (117374 ms total) 2023-11-25T05:07:32.992 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ PASSED ] 16 tests. 2023-11-25T05:07:33.949 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[93494]: starting. 2023-11-25T05:07:33.949 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[93494]: creating pool ceph_test_rados_open_pools_parallel.smithi007-93450 2023-11-25T05:07:33.949 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[93494]: created object 0... 2023-11-25T05:07:33.949 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[93494]: created object 25... 2023-11-25T05:07:33.949 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[93494]: created object 49... 2023-11-25T05:07:33.949 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[93494]: finishing. 2023-11-25T05:07:33.949 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[93494]: shutting down. 2023-11-25T05:07:33.949 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_2_[93495]: starting. 2023-11-25T05:07:33.949 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_2_[93495]: rados_pool_create. 2023-11-25T05:07:33.950 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_2_[93495]: rados_ioctx_create. 2023-11-25T05:07:33.950 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_2_[93495]: shutting down. 2023-11-25T05:07:33.950 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: ******************************* 2023-11-25T05:07:33.950 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[110486]: starting. 2023-11-25T05:07:33.950 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[110486]: creating pool ceph_test_rados_open_pools_parallel.smithi007-93450 2023-11-25T05:07:33.950 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[110486]: created object 0... 2023-11-25T05:07:33.950 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[110486]: created object 25... 2023-11-25T05:07:33.950 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[110486]: created object 49... 2023-11-25T05:07:33.950 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[110486]: finishing. 2023-11-25T05:07:33.950 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[110486]: shutting down. 2023-11-25T05:07:33.950 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: ******************************* 2023-11-25T05:07:33.950 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_4_[110487]: starting. 2023-11-25T05:07:33.951 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_4_[110487]: rados_pool_create. 2023-11-25T05:07:33.951 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_4_[110487]: rados_ioctx_create. 2023-11-25T05:07:33.951 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_4_[110487]: shutting down. 2023-11-25T05:07:33.951 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: ******************************* 2023-11-25T05:07:33.951 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: ******************************* 2023-11-25T05:07:33.951 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: ******* SUCCESS ********** 2023-11-25T05:07:34.765 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2023-11-25T05:07:34.765 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2023-11-25T05:07:34.766 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2023-11-25T05:07:34.766 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2023-11-25T05:07:34.766 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2023-11-25T05:07:34.766 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify 2023-11-25T05:07:34.766 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (55690 ms) 2023-11-25T05:07:34.766 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2023-11-25T05:07:34.766 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (5169 ms) 2023-11-25T05:07:34.766 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (60859 ms total) 2023-11-25T05:07:34.766 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: 2023-11-25T05:07:34.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2023-11-25T05:07:34.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2023-11-25T05:07:34.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify 2023-11-25T05:07:34.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (6109 ms) 2023-11-25T05:07:34.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2023-11-25T05:07:34.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify 2023-11-25T05:07:34.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (35934 ms) 2023-11-25T05:07:34.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2023-11-25T05:07:34.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (258 ms) 2023-11-25T05:07:34.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2023-11-25T05:07:34.767 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (166 ms) 2023-11-25T05:07:34.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2023-11-25T05:07:34.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94216741980672 notify_id 403726925824 notifier_gid 4610 2023-11-25T05:07:34.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (467 ms) 2023-11-25T05:07:34.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2023-11-25T05:07:34.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94216741980672 notify_id 403726925828 notifier_gid 4610 2023-11-25T05:07:34.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (222 ms) 2023-11-25T05:07:34.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2023-11-25T05:07:34.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94216743626688 notify_id 403726925825 notifier_gid 4610 2023-11-25T05:07:34.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (387 ms) 2023-11-25T05:07:34.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2023-11-25T05:07:34.768 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94216744383696 notify_id 408021893125 notifier_gid 4610 2023-11-25T05:07:34.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (1381 ms) 2023-11-25T05:07:34.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2023-11-25T05:07:34.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94216744409760 notify_id 408021893121 notifier_gid 4610 2023-11-25T05:07:34.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (262 ms) 2023-11-25T05:07:34.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2023-11-25T05:07:34.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94216744409760 notify_id 408021893126 notifier_gid 4610 2023-11-25T05:07:34.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (282 ms) 2023-11-25T05:07:34.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2023-11-25T05:07:34.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: trying... 2023-11-25T05:07:34.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94216744409760 notify_id 408021893122 notifier_gid 4610 2023-11-25T05:07:34.769 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: timed out 2023-11-25T05:07:37.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushiasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi007-92726-16"}]: dispatch 2023-11-25T05:07:37.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:20.057249+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi007-92842-10"}]: dispatch 2023-11-25T05:07:37.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:20.410645+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi007-93181-6", "mode": "writeback"}]: dispatch 2023-11-25T05:07:37.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:20.444332+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi007-93106-7"}]': finished 2023-11-25T05:07:37.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:21.497354+0000 mon.a [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-11-25T05:07:37.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:21.838349+0000 mon.c [INF] from='client.? 172.21.15.7:0/1375230274' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi007-93164-12"}]: dispatch 2023-11-25T05:07:37.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:22.953318+0000 mon.c [INF] from='client.? 172.21.15.7:0/2704597848' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-11-25T05:07:37.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:23.176162+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi007-93164-12"}]: dispatch 2023-11-25T05:07:37.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:23.197337+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi007-92842-10"}]': finished 2023-11-25T05:07:37.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:23.199866+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi007-93181-6", "mode": "writeback"}]': finished 2023-11-25T05:07:37.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:24.807518+0000 client.admin [INF] threexx 2023-11-25T05:07:37.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2023-11-25T05:07:24.852110+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-11-25T05:07:37.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (35057 ms) 2023-11-25T05:07:37.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (121332 ms total) 2023-11-25T05:07:37.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: 2023-11-25T05:07:37.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [----------] Global test environment tear-down 2023-11-25T05:07:37.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (121332 ms total) 2023-11-25T05:07:37.112 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ PASSED ] 4 tests. 2023-11-25T05:07:38.219 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[93509]: starting. 2023-11-25T05:07:38.219 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[93509]: creating pool ceph_test_rados_list_parallel.smithi007-93469 2023-11-25T05:07:38.219 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[93509]: created object 0... 2023-11-25T05:07:38.219 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[93509]: created object 25... 2023-11-25T05:07:38.219 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[93509]: created object 49... 2023-11-25T05:07:38.219 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[93509]: finishing. 2023-11-25T05:07:38.220 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[93509]: shutting down. 2023-11-25T05:07:38.220 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[93510]: starting. 2023-11-25T05:07:38.220 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[93510]: listing objects. 2023-11-25T05:07:38.220 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[93510]: listed object 0... 2023-11-25T05:07:38.220 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[93510]: listed object 25... 2023-11-25T05:07:38.220 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[93510]: saw 50 objects 2023-11-25T05:07:38.220 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[93510]: shutting down. 2023-11-25T05:07:38.220 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:38.220 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[111299]: starting. 2023-11-25T05:07:38.220 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[111299]: creating pool ceph_test_rados_list_parallel.smithi007-93469 2023-11-25T05:07:38.220 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[111299]: created object 0... 2023-11-25T05:07:38.220 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[111299]: created object 25... 2023-11-25T05:07:38.221 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[111299]: created object 49... 2023-11-25T05:07:38.221 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[111299]: finishing. 2023-11-25T05:07:38.221 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[111299]: shutting down. 2023-11-25T05:07:38.221 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:38.221 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[111300]: starting. 2023-11-25T05:07:38.221 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[111300]: listing objects. 2023-11-25T05:07:38.221 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[111300]: listed object 0... 2023-11-25T05:07:38.221 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[111300]: listed object 25... 2023-11-25T05:07:38.221 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[111300]: saw 46 objects 2023-11-25T05:07:38.221 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[111300]: shutting down. 2023-11-25T05:07:38.221 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:38.221 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[111301]: starting. 2023-11-25T05:07:38.222 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[111301]: removed 25 objects... 2023-11-25T05:07:38.222 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[111301]: removed half of the objects 2023-11-25T05:07:38.222 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[111301]: removed 50 objects... 2023-11-25T05:07:38.223 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[111301]: removed 50 objects 2023-11-25T05:07:38.223 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[111301]: shutting down. 2023-11-25T05:07:38.223 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:38.223 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:38.223 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[119662]: starting. 2023-11-25T05:07:38.224 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[119662]: creating pool ceph_test_rados_list_parallel.smithi007-93469 2023-11-25T05:07:38.224 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[119662]: created object 0... 2023-11-25T05:07:38.224 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[119662]: created object 25... 2023-11-25T05:07:38.224 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[119662]: created object 49... 2023-11-25T05:07:38.224 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[119662]: finishing. 2023-11-25T05:07:38.224 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[119662]: shutting down. 2023-11-25T05:07:38.224 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:38.224 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:38.224 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[119664]: starting. 2023-11-25T05:07:38.224 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[119664]: listing objects. 2023-11-25T05:07:38.224 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[119664]: listed object 0... 2023-11-25T05:07:38.224 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[119664]: listed object 25... 2023-11-25T05:07:38.224 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[119664]: listed object 50... 2023-11-25T05:07:38.225 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[119664]: saw 54 objects 2023-11-25T05:07:38.225 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[119664]: shutting down. 2023-11-25T05:07:38.225 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:38.225 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:38.225 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[119665]: starting. 2023-11-25T05:07:38.225 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[119665]: added 25 objects... 2023-11-25T05:07:38.225 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[119665]: added half of the objects 2023-11-25T05:07:38.225 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[119665]: added 50 objects... 2023-11-25T05:07:38.225 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[119665]: added 50 objects 2023-11-25T05:07:38.225 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[119665]: shutting down. 2023-11-25T05:07:38.225 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:38.225 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:39.786 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[93467]: starting. 2023-11-25T05:07:39.786 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[93467]: creating pool ceph_test_rados_delete_pools_parallel.smithi007-93417 2023-11-25T05:07:39.787 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[93467]: created object 0... 2023-11-25T05:07:39.787 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[93467]: created object 25... 2023-11-25T05:07:39.787 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[93467]: created object 49... 2023-11-25T05:07:39.787 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[93467]: finishing. 2023-11-25T05:07:39.787 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[93467]: shutting down. 2023-11-25T05:07:39.787 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_2_[93468]: starting. 2023-11-25T05:07:39.787 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_2_[93468]: deleting pool ceph_test_rados_delete_pools_parallel.smithi007-93417 2023-11-25T05:07:39.787 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_2_[93468]: shutting down. 2023-11-25T05:07:39.787 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******************************* 2023-11-25T05:07:39.787 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[110495]: starting. 2023-11-25T05:07:39.787 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[110495]: creating pool ceph_test_rados_delete_pools_parallel.smithi007-93417 2023-11-25T05:07:39.787 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[110495]: created object 0... 2023-11-25T05:07:39.787 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[110495]: created object 25... 2023-11-25T05:07:39.788 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[110495]: created object 49... 2023-11-25T05:07:39.788 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[110495]: finishing. 2023-11-25T05:07:39.788 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[110495]: shutting down. 2023-11-25T05:07:39.788 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******************************* 2023-11-25T05:07:39.788 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[110497]: starting. 2023-11-25T05:07:39.788 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[110497]: listing objects. 2023-11-25T05:07:39.788 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[110497]: listed object 0... 2023-11-25T05:07:39.788 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[110497]: listed object 25... 2023-11-25T05:07:39.788 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[110497]: saw 50 objects 2023-11-25T05:07:39.788 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[110497]: shutting down. 2023-11-25T05:07:39.788 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******************************* 2023-11-25T05:07:39.788 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_4_[110496]: starting. 2023-11-25T05:07:39.788 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_4_[110496]: deleting pool ceph_test_rados_delete_pools_parallel.smithi007-93417 2023-11-25T05:07:39.789 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_4_[110496]: shutting down. 2023-11-25T05:07:39.789 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******************************* 2023-11-25T05:07:39.789 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******************************* 2023-11-25T05:07:39.789 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******* SUCCESS ********** 2023-11-25T05:07:40.095 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: Running main() from gmock_main.cc 2023-11-25T05:07:40.095 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2023-11-25T05:07:40.095 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] Global test environment set-up. 2023-11-25T05:07:40.095 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2023-11-25T05:07:40.095 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: seed 92746 2023-11-25T05:07:40.096 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2023-11-25T05:07:40.096 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2023-11-25T05:07:40.096 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2023-11-25T05:07:40.096 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (37942 ms) 2023-11-25T05:07:40.096 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2023-11-25T05:07:40.096 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (13509 ms) 2023-11-25T05:07:40.096 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2023-11-25T05:07:40.096 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (975 ms) 2023-11-25T05:07:40.096 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2023-11-25T05:07:40.096 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (164 ms) 2023-11-25T05:07:40.096 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2023-11-25T05:07:40.096 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (57 ms) 2023-11-25T05:07:40.096 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2023-11-25T05:07:40.097 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (8642 ms) 2023-11-25T05:07:40.097 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2023-11-25T05:07:40.097 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (3446 ms) 2023-11-25T05:07:40.097 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2023-11-25T05:07:40.097 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (8078 ms) 2023-11-25T05:07:40.097 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2023-11-25T05:07:40.097 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (1669 ms) 2023-11-25T05:07:40.097 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2023-11-25T05:07:40.097 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (968 ms) 2023-11-25T05:07:40.097 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2023-11-25T05:07:40.097 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (107 ms) 2023-11-25T05:07:40.097 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2023-11-25T05:07:40.098 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (684 ms) 2023-11-25T05:07:40.098 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2023-11-25T05:07:40.098 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (190 ms) 2023-11-25T05:07:40.098 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2023-11-25T05:07:40.098 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (125 ms) 2023-11-25T05:07:40.098 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2023-11-25T05:07:40.098 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (558 ms) 2023-11-25T05:07:40.098 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-11-25T05:07:40.098 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (246 ms) 2023-11-25T05:07:40.098 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2023-11-25T05:07:40.098 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (128 ms) 2023-11-25T05:07:40.098 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2023-11-25T05:07:40.098 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (85 ms) 2023-11-25T05:07:40.099 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2023-11-25T05:07:40.099 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (131 ms) 2023-11-25T05:07:40.099 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (77705 ms total) 2023-11-25T05:07:40.099 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: 2023-11-25T05:07:40.099 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2023-11-25T05:07:40.099 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2023-11-25T05:07:40.099 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (7261 ms) 2023-11-25T05:07:40.099 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2023-11-25T05:07:40.099 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (845 ms) 2023-11-25T05:07:40.099 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2023-11-25T05:07:40.100 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (782 ms) 2023-11-25T05:07:40.100 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2023-11-25T05:07:40.100 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (473 ms) 2023-11-25T05:07:40.100 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2023-11-25T05:07:53.185 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 list_parallel: ******************************* 2023-11-25T05:07:53.186 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[129423]: starting. 2023-11-25T05:07:53.186 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[129423]: creating pool ceph_test_rados_list_parallel.smithi007-93469 2023-11-25T05:07:53.186 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[129423]: created object 0... 2023-11-25T05:07:53.186 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[129423]: created object 25... 2023-11-25T05:07:53.186 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[129423]: created object 49... 2023-11-25T05:07:53.186 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[129423]: finishing. 2023-11-25T05:07:53.186 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[129423]: shutting down. 2023-11-25T05:07:53.186 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.186 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.186 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.187 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[129424]: starting. 2023-11-25T05:07:53.187 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[129424]: listing objects. 2023-11-25T05:07:53.187 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[129424]: listed object 0... 2023-11-25T05:07:53.187 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[129424]: listed object 25... 2023-11-25T05:07:53.187 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[129424]: listed object 50... 2023-11-25T05:07:53.187 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[129424]: listed object 75... 2023-11-25T05:07:53.187 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[129424]: listed object 100... 2023-11-25T05:07:53.187 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[129424]: saw 104 objects 2023-11-25T05:07:53.187 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[129424]: shutting down. 2023-11-25T05:07:53.187 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.187 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.187 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.187 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[129426]: starting. 2023-11-25T05:07:53.188 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[129426]: added 25 objects... 2023-11-25T05:07:53.188 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[129426]: added half of the objects 2023-11-25T05:07:53.188 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[129426]: added 50 objects... 2023-11-25T05:07:53.188 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[129426]: added 50 objects 2023-11-25T05:07:53.188 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[129426]: shutting down. 2023-11-25T05:07:53.188 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.188 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.188 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.188 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[129425]: starting. 2023-11-25T05:07:53.188 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[129425]: added 25 objects... 2023-11-25T05:07:53.188 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[129425]: added half of the objects 2023-11-25T05:07:53.188 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[129425]: added 50 objects... 2023-11-25T05:07:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[129425]: added 50 objects 2023-11-25T05:07:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[129425]: shutting down. 2023-11-25T05:07:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[129427]: starting. 2023-11-25T05:07:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[129427]: removed 25 objects... 2023-11-25T05:07:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[129427]: removed half of the objects 2023-11-25T05:07:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[129427]: removed 50 objects... 2023-11-25T05:07:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[129427]: removed 50 objects 2023-11-25T05:07:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[129427]: shutting down. 2023-11-25T05:07:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[142322]: starting. 2023-11-25T05:07:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[142322]: creating pool ceph_test_rados_list_parallel.smithi007-93469 2023-11-25T05:07:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[142322]: created object 0... 2023-11-25T05:07:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[142322]: created object 25... 2023-11-25T05:07:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[142322]: created object 49... 2023-11-25T05:07:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[142322]: finishing. 2023-11-25T05:07:53.191 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[142322]: shutting down. 2023-11-25T05:07:53.191 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.191 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.191 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.191 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:07:53.191 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[142326]: starting. 2023-11-25T05:07:56.439 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:38.637 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:56.447 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:38.644 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:56.740 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:04:38.884 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:56.748 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:04:38.892 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:56.877 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:04:39.021 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:56.901 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:39.099 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:57.180 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:39.378 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:57.188 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:39.386 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:07:57.375 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:39.573 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:00.552 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[142326]: listing objects. 2023-11-25T05:08:00.552 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[142326]: listed object 0... 2023-11-25T05:08:00.552 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[142326]: listed object 25... 2023-11-25T05:08:00.553 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[142326]: listed object 50... 2023-11-25T05:08:00.553 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[142326]: listed object 75... 2023-11-25T05:08:00.553 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[142326]: listed object 100... 2023-11-25T05:08:00.553 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[142326]: listed object 125... 2023-11-25T05:08:00.554 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[142326]: saw 150 objects 2023-11-25T05:08:00.554 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[142326]: shutting down. 2023-11-25T05:08:00.554 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:08:00.554 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:08:00.554 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:08:00.555 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:08:00.555 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[142327]: starting. 2023-11-25T05:08:00.555 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[142327]: added 25 objects... 2023-11-25T05:08:00.556 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[142327]: added half of the objects 2023-11-25T05:08:00.556 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[142327]: added 50 objects... 2023-11-25T05:08:00.556 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[142327]: added 50 objects 2023-11-25T05:08:00.556 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[142327]: shutting down. 2023-11-25T05:08:00.556 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:08:00.557 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:08:00.557 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:08:00.557 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:08:00.557 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2023-11-25T05:08:00.557 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******* SUCCESS ********** 2023-11-25T05:08:01.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: Running main() from gmock_main.cc 2023-11-25T05:08:01.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2023-11-25T05:08:01.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] Global test environment set-up. 2023-11-25T05:08:01.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2023-11-25T05:08:01.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: seed 92898 2023-11-25T05:08:01.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2023-11-25T05:08:01.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (28344 ms) 2023-11-25T05:08:01.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2023-11-25T05:08:01.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (17362 ms) 2023-11-25T05:08:01.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2023-11-25T05:08:01.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (9302 ms) 2023-11-25T05:08:01.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2023-11-25T05:08:01.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (3407 ms) 2023-11-25T05:08:01.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2023-11-25T05:08:01.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (3447 ms) 2023-11-25T05:08:01.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2023-11-25T05:08:01.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (4045 ms) 2023-11-25T05:08:01.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2023-11-25T05:08:01.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (6843 ms) 2023-11-25T05:08:01.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2023-11-25T05:08:01.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (1965 ms) 2023-11-25T05:08:01.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (74715 ms total) 2023-11-25T05:08:01.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: 2023-11-25T05:08:01.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2023-11-25T05:08:01.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2023-11-25T05:08:01.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (8600 ms) 2023-11-25T05:08:01.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2023-11-25T05:08:01.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (151 ms) 2023-11-25T05:08:01.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2023-11-25T05:08:01.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (2165 ms) 2023-11-25T05:08:01.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2023-11-25T05:08:01.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1272 ms) 2023-11-25T05:08:01.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2023-11-25T05:08:01.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (268 ms) 2023-11-25T05:08:01.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2023-11-25T05:08:01.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (475 ms) 2023-11-25T05:08:01.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2023-11-25T05:08:01.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (1277 ms) 2023-11-25T05:08:01.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2023-11-25T05:08:01.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (224 ms) 2023-11-25T05:08:01.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (14432 ms total) 2023-11-25T05:08:01.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: 2023-11-25T05:08:01.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] Global test environment tear-down 2023-11-25T05:08:01.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (145844 ms total) 2023-11-25T05:08:01.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ PASSED ] 16 tests. 2023-11-25T05:08:01.491 INFO:tasks.workunit.client.0.smithi007.stdout: (1070 ms) 2023-11-25T05:08:01.491 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2023-11-25T05:08:01.491 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (290 ms) 2023-11-25T05:08:01.492 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2023-11-25T05:08:01.492 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (226 ms) 2023-11-25T05:08:01.492 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2023-11-25T05:08:01.492 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (155 ms) 2023-11-25T05:08:01.492 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2023-11-25T05:08:01.492 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (411 ms) 2023-11-25T05:08:01.492 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2023-11-25T05:08:01.492 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (266 ms) 2023-11-25T05:08:01.492 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2023-11-25T05:08:01.492 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (404 ms) 2023-11-25T05:08:01.492 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2023-11-25T05:08:01.492 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (260 ms) 2023-11-25T05:08:01.493 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2023-11-25T05:08:01.493 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (1640 ms) 2023-11-25T05:08:01.493 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2023-11-25T05:08:01.493 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (601 ms) 2023-11-25T05:08:01.493 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2023-11-25T05:08:01.493 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (301 ms) 2023-11-25T05:08:01.493 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2023-11-25T05:08:01.493 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (125 ms) 2023-11-25T05:08:01.493 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2023-11-25T05:08:01.493 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (184 ms) 2023-11-25T05:08:01.493 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (15294 ms total) 2023-11-25T05:08:01.493 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: 2023-11-25T05:08:01.493 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] Global test environment tear-down 2023-11-25T05:08:01.494 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (145927 ms total) 2023-11-25T05:08:01.494 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ PASSED ] 37 tests. 2023-11-25T05:08:02.054 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: Running main() from gmock_main.cc 2023-11-25T05:08:02.054 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2023-11-25T05:08:02.054 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] Global test environment set-up. 2023-11-25T05:08:02.054 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] 14 tests from LibRadosIo 2023-11-25T05:08:02.054 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2023-11-25T05:08:02.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (47370 ms) 2023-11-25T05:08:02.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2023-11-25T05:08:02.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2023-11-25T05:08:02.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2023-11-25T05:08:02.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: no timeout :/ 2023-11-25T05:08:02.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: no timeout :/ 2023-11-25T05:08:02.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: no timeout :/ 2023-11-25T05:08:02.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: no timeout :/ 2023-11-25T05:08:02.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (6905 ms) 2023-11-25T05:08:02.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2023-11-25T05:08:02.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (2887 ms) 2023-11-25T05:08:02.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2023-11-25T05:08:02.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.Checksum (2753 ms) 2023-11-25T05:08:02.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2023-11-25T05:08:02.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (2041 ms) 2023-11-25T05:08:02.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2023-11-25T05:08:02.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (2688 ms) 2023-11-25T05:08:02.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2023-11-25T05:08:02.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (4051 ms) 2023-11-25T05:08:02.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2023-11-25T05:08:02.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (3037 ms) 2023-11-25T05:08:02.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2023-11-25T05:08:02.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.TruncTest (162 ms) 2023-11-25T05:08:02.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2023-11-25T05:08:02.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (944 ms) 2023-11-25T05:08:02.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2023-11-25T05:08:02.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (92 ms) 2023-11-25T05:08:02.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2023-11-25T05:08:02.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.RmXattr (1999 ms) 2023-11-25T05:08:02.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2023-11-25T05:08:02.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.XattrIter (179 ms) 2023-11-25T05:08:02.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] 14 tests from LibRadosIo (75109 ms total) 2023-11-25T05:08:02.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: 2023-11-25T05:08:02.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2023-11-25T05:08:02.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2023-11-25T05:08:02.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (5199 ms) 2023-11-25T05:08:02.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2023-11-25T05:08:02.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (2395 ms) 2023-11-25T05:08:02.058 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2023-11-25T05:08:02.058 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (269 ms) 2023-11-25T05:08:02.058 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2023-11-25T05:08:02.058 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (378 ms) 2023-11-25T05:08:02.058 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2023-11-25T05:08:02.058 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (310 ms) 2023-11-25T05:08:02.058 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2023-11-25T05:08:02.058 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (163 ms) 2023-11-25T05:08:02.058 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2023-11-25T05:08:02.058 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (418 ms) 2023-11-25T05:08:02.058 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2023-11-25T05:08:02.058 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (1314 ms) 2023-11-25T05:08:02.059 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2023-11-25T05:08:02.059 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (628 ms) 2023-11-25T05:08:02.059 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2023-11-25T05:08:02.059 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (461 ms) 2023-11-25T05:08:02.059 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] 10 tests from LibRadosIoEC (11535 ms total) 2023-11-25T05:08:02.059 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: 2023-11-25T05:08:02.059 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] Global test environment tear-down 2023-11-25T05:08:02.060 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [==========] 24 tests from 2 test suites ran. (146502 ms total) 2023-11-25T05:08:02.060 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ PASSED ] 24 tests. 2023-11-25T05:08:02.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: Running main() from gmock_main.cc 2023-11-25T05:08:02.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2023-11-25T05:08:02.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] Global test environment set-up. 2023-11-25T05:08:02.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2023-11-25T05:08:02.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2023-11-25T05:08:02.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify_test_cb 2023-11-25T05:08:02.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (31890 ms) 2023-11-25T05:08:02.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2023-11-25T05:08:02.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-11-25T05:08:02.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94006160017920 err -107 2023-11-25T05:08:02.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (12830 ms) 2023-11-25T05:08:02.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2023-11-25T05:08:02.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-11-25T05:08:02.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94006160017920 err -107 2023-11-25T05:08:02.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (12682 ms) 2023-11-25T05:08:02.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2023-11-25T05:08:02.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 309237645312 cookie 94006161715904 2023-11-25T05:08:02.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (1879 ms) 2023-11-25T05:08:02.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2023-11-25T05:08:02.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 313532612609 cookie 94006161715904 2023-11-25T05:08:02.161 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (4771 ms) 2023-11-25T05:08:02.161 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2023-11-25T05:08:02.161 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 352187318274 cookie 94006162413824 2023-11-25T05:08:02.161 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (7907 ms) 2023-11-25T05:08:02.161 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2023-11-25T05:08:02.161 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 360777252864 cookie 94006162413824 2023-11-25T05:08:02.161 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 360777252864 cookie 94006162430688 2023-11-25T05:08:02.161 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (3783 ms) 2023-11-25T05:08:02.161 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2023-11-25T05:08:02.161 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 360777252864 cookie 94006162430688 2023-11-25T05:08:02.161 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 360777252865 cookie 94006162430688 2023-11-25T05:08:02.161 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3835 ms) 2023-11-25T05:08:02.162 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2023-11-25T05:08:02.162 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2023-11-25T05:08:02.162 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94006162430688 err -107 2023-11-25T05:08:02.162 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 373662154755 cookie 94006162430688 2023-11-25T05:08:02.162 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (8452 ms) 2023-11-25T05:08:02.162 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2023-11-25T05:08:02.162 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2023-11-25T05:08:02.162 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94006162413824 err -107 2023-11-25T05:08:02.162 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1341 ms) 2023-11-25T05:08:02.162 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (89370 ms total) 2023-11-25T05:08:02.162 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: 2023-11-25T05:08:02.162 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2023-11-25T05:08:02.163 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2023-11-25T05:08:02.163 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify_test_cb 2023-11-25T05:08:02.163 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (6687 ms) 2023-11-25T05:08:02.163 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (6687 ms total) 2023-11-25T05:08:02.163 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: 2023-11-25T05:08:02.163 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] Global test environment tear-down 2023-11-25T05:08:02.163 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (146404 ms total) 2023-11-25T05:08:02.164 INFO:tasks.workunit.client.0.smithi007.stdout: api_watchd846d26fb0 2023-11-25T05:08:02.164 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d846d27a20 2023-11-25T05:08:02.164 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d846d261a0 2023-11-25T05:08:02.164 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d846d28870 2023-11-25T05:08:02.164 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d846d277f0 2023-11-25T05:08:02.164 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d846d29780 2023-11-25T05:08:02.164 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d846d28640 2023-11-25T05:08:02.164 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d846d2a600 2023-11-25T05:08:02.164 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d846d29490 2023-11-25T05:08:02.164 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8475fdd00 2023-11-25T05:08:02.164 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d8475fe7c0 2023-11-25T05:08:02.165 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x55d846d2a3f0 2023-11-25T05:08:02.165 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: done. 2023-11-25T05:08:02.165 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (37425 ms) 2023-11-25T05:08:02.165 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (37425 ms total) 2023-11-25T05:08:02.165 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2023-11-25T05:08:02.165 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2023-11-25T05:08:02.165 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2023-11-25T05:08:02.165 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2023-11-25T05:08:02.165 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2023-11-25T05:08:02.165 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.Exec (7881 ms) 2023-11-25T05:08:02.165 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2023-11-25T05:08:02.165 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (13442 ms) 2023-11-25T05:08:02.166 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2023-11-25T05:08:02.166 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (1230 ms) 2023-11-25T05:08:02.166 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2023-11-25T05:08:02.166 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5743 ms) 2023-11-25T05:08:02.166 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2023-11-25T05:08:02.166 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2023-11-25T05:08:02.166 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2023-11-25T05:08:02.166 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2023-11-25T05:08:02.166 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2023-11-25T05:08:02.166 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (74364 ms) 2023-11-25T05:08:02.166 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 8 tests from LibRadosMisc (102660 ms total) 2023-11-25T05:08:02.166 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2023-11-25T05:08:02.166 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] Global test environment tear-down 2023-11-25T05:08:02.167 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (146525 ms total) 2023-11-25T05:08:02.167 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ PASSED ] 12 tests. 2023-11-25T05:08:02.167 INFO:tasks.workunit.client.0.smithi007.stdout:_notify: [ PASSED ] 11 tests. 2023-11-25T05:08:03.061 INFO:tasks.workunit.client.0.smithi007.stdout:ng 2023-11-25T05:08:03.061 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushed 2023-11-25T05:08:03.061 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3133 ms) 2023-11-25T05:08:03.061 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2023-11-25T05:08:03.061 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: trying... 2023-11-25T05:08:03.061 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94216744413968 notify_id 416611827713 notifier_gid 4610 2023-11-25T05:08:03.061 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: timed out 2023-11-25T05:08:03.061 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushing 2023-11-25T05:08:03.061 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushed 2023-11-25T05:08:03.061 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3888 ms) 2023-11-25T05:08:03.061 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2023-11-25T05:08:03.062 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: List watches 2023-11-25T05:08:03.062 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify2 2023-11-25T05:08:03.062 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94216744415008 notify_id 420906795010 notifier_gid 4610 2023-11-25T05:08:03.062 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify2 done 2023-11-25T05:08:03.062 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: watch_check 2023-11-25T05:08:03.062 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: unwatch2 2023-11-25T05:08:03.062 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushing 2023-11-25T05:08:03.062 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: done 2023-11-25T05:08:03.062 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (4510 ms) 2023-11-25T05:08:03.062 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2023-11-25T05:08:03.062 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: List watches 2023-11-25T05:08:03.062 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify2 2023-11-25T05:08:03.063 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94216744415008 notify_id 425201762307 notifier_gid 4610 2023-11-25T05:08:03.063 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify2 done 2023-11-25T05:08:03.063 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: watch_check 2023-11-25T05:08:03.063 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: unwatch2 2023-11-25T05:08:03.063 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushing 2023-11-25T05:08:03.063 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: done 2023-11-25T05:08:03.063 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3455 ms) 2023-11-25T05:08:03.063 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (60455 ms total) 2023-11-25T05:08:03.063 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: 2023-11-25T05:08:03.063 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2023-11-25T05:08:03.063 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (147296 ms total) 2023-11-25T05:08:03.063 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2023-11-25T05:08:36.108 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:18.306 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:36.181 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:18.378 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:36.183 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:05:18.327 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:36.186 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:05:18.329 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:36.203 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:18.401 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:36.257 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:18.455 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:36.277 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:05:18.421 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:36.364 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:18.562 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:36.417 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:18.615 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:50.185 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: Running main() from gmock_main.cc 2023-11-25T05:08:50.185 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2023-11-25T05:08:50.185 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] Global test environment set-up. 2023-11-25T05:08:50.185 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2023-11-25T05:08:50.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2023-11-25T05:08:50.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (39804 ms) 2023-11-25T05:08:50.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2023-11-25T05:08:50.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (6167 ms) 2023-11-25T05:08:50.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2023-11-25T05:08:50.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (18056 ms) 2023-11-25T05:08:50.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2023-11-25T05:08:50.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (5941 ms) 2023-11-25T05:08:50.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (69968 ms total) 2023-11-25T05:08:50.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: 2023-11-25T05:08:50.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2023-11-25T05:08:50.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2023-11-25T05:08:50.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (25960 ms) 2023-11-25T05:08:50.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2023-11-25T05:08:50.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (27631 ms) 2023-11-25T05:08:50.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2023-11-25T05:08:50.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (6140 ms) 2023-11-25T05:08:50.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (59731 ms total) 2023-11-25T05:08:50.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: 2023-11-25T05:08:50.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2023-11-25T05:08:50.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2023-11-25T05:08:50.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3014 ms) 2023-11-25T05:08:50.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2023-11-25T05:08:50.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2953 ms) 2023-11-25T05:08:50.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2023-11-25T05:08:50.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (3178 ms) 2023-11-25T05:08:50.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2023-11-25T05:08:50.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1837 ms) 2023-11-25T05:08:50.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (10982 ms total) 2023-11-25T05:08:50.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: 2023-11-25T05:08:50.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2023-11-25T05:08:50.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2023-11-25T05:08:50.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (7896 ms) 2023-11-25T05:08:50.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2023-11-25T05:08:50.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (7798 ms) 2023-11-25T05:08:50.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (15694 ms total) 2023-11-25T05:08:50.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: 2023-11-25T05:08:50.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] Global test environment tear-down 2023-11-25T05:08:50.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (194487 ms total) 2023-11-25T05:08:50.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ PASSED ] 13 tests. 2023-11-25T05:08:56.136 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:38.334 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:56.209 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:38.405 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:56.217 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:05:38.361 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:56.226 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:05:38.370 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:56.270 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:38.467 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:56.273 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:38.471 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:56.305 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:05:38.448 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:56.318 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:38.516 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:08:56.348 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:38.546 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:05.382 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2023-11-25T05:09:05.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [==========] Running 19 tests from 4 test suites. 2023-11-25T05:09:05.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2023-11-25T05:09:05.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2023-11-25T05:09:05.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: seed 93058 2023-11-25T05:09:05.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2023-11-25T05:09:05.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (12329 ms) 2023-11-25T05:09:05.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2023-11-25T05:09:05.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (7866 ms) 2023-11-25T05:09:05.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2023-11-25T05:09:05.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (8044 ms) 2023-11-25T05:09:05.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2023-11-25T05:09:05.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (25071 ms) 2023-11-25T05:09:05.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2023-11-25T05:09:05.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (10770 ms) 2023-11-25T05:09:05.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (64080 ms total) 2023-11-25T05:09:05.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2023-11-25T05:09:05.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2023-11-25T05:09:05.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2023-11-25T05:09:05.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (32626 ms) 2023-11-25T05:09:05.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2023-11-25T05:09:05.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (38049 ms) 2023-11-25T05:09:05.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2023-11-25T05:09:05.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (16205 ms) 2023-11-25T05:09:05.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2023-11-25T05:09:05.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (5105 ms) 2023-11-25T05:09:05.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2023-11-25T05:09:05.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (3693 ms) 2023-11-25T05:09:05.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-11-25T05:09:05.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7506-gec928368/rpm/el9/BUILD/ceph-18.0.0-7506-gec928368/src/test/librados/snapshots_cxx.cc:463: Skipped 2023-11-25T05:09:05.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2023-11-25T05:09:05.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2023-11-25T05:09:05.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2023-11-25T05:09:05.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi007-93058-7 2023-11-25T05:09:05.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: waiting for snaps to purge 2023-11-25T05:09:05.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (19166 ms) 2023-11-25T05:09:05.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (114844 ms total) 2023-11-25T05:09:05.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2023-11-25T05:09:05.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2023-11-25T05:09:05.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2023-11-25T05:09:05.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (4814 ms) 2023-11-25T05:09:05.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2023-11-25T05:09:05.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2084 ms) 2023-11-25T05:09:05.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2023-11-25T05:09:05.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (3132 ms) 2023-11-25T05:09:05.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2023-11-25T05:09:05.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2949 ms) 2023-11-25T05:09:05.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (12979 ms total) 2023-11-25T05:09:05.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2023-11-25T05:09:05.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2023-11-25T05:09:14.458 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN api_aio: Running main() from gmock_main.cc 2023-11-25T05:09:14.458 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2023-11-25T05:09:14.458 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] Global test environment set-up. 2023-11-25T05:09:14.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] 26 tests from LibRadosAio 2023-11-25T05:09:14.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2023-11-25T05:09:14.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.TooBig (4413 ms) 2023-11-25T05:09:14.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2023-11-25T05:09:14.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (22700 ms) 2023-11-25T05:09:14.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2023-11-25T05:09:14.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (24222 ms) 2023-11-25T05:09:14.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2023-11-25T05:09:14.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (11186 ms) 2023-11-25T05:09:14.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2023-11-25T05:09:14.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (24627 ms) 2023-11-25T05:09:14.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2023-11-25T05:09:14.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (25121 ms) 2023-11-25T05:09:14.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2023-11-25T05:09:14.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (21873 ms) 2023-11-25T05:09:14.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2023-11-25T05:09:14.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (15540 ms) 2023-11-25T05:09:14.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2023-11-25T05:09:14.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3471 ms) 2023-11-25T05:09:14.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2023-11-25T05:09:14.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (4035 ms) 2023-11-25T05:09:14.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2023-11-25T05:09:14.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (5121 ms) 2023-11-25T05:09:14.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2023-11-25T05:09:14.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3923 ms) 2023-11-25T05:09:14.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2023-11-25T05:09:14.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3272 ms) 2023-11-25T05:09:14.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2023-11-25T05:09:14.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2785 ms) 2023-11-25T05:09:14.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2023-11-25T05:09:14.464 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.Flush (3133 ms) 2023-11-25T05:09:14.464 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2023-11-25T05:09:14.464 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3446 ms) 2023-11-25T05:09:14.464 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2023-11-25T05:09:14.464 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (5165 ms) 2023-11-25T05:09:14.464 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2023-11-25T05:09:14.464 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (4719 ms) 2023-11-25T05:09:14.465 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2023-11-25T05:09:14.465 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3747 ms) 2023-11-25T05:09:14.465 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2023-11-25T05:09:14.465 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (4245 ms) 2023-11-25T05:09:14.465 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2023-11-25T05:09:14.466 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3104 ms) 2023-11-25T05:09:14.466 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2023-11-25T05:09:14.466 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (4074 ms) 2023-11-25T05:09:14.466 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2023-11-25T05:09:14.466 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3430 ms) 2023-11-25T05:09:14.467 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2023-11-25T05:09:14.467 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3520 ms) 2023-11-25T05:09:14.467 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2023-11-25T05:09:14.470 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (4600 ms) 2023-11-25T05:09:14.471 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2023-11-25T05:09:14.471 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3447 ms) 2023-11-25T05:09:14.471 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] 26 tests from LibRadosAio (218919 ms total) 2023-11-25T05:09:14.471 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: 2023-11-25T05:09:14.472 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2023-11-25T05:09:16.125 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:58.323 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:16.125 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:58.323 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:16.134 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:58.332 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:16.173 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:05:58.317 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:16.176 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:05:58.320 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:16.183 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:58.381 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:16.189 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:58.387 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:16.224 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:05:58.367 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:16.244 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:58.442 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:22.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.Simpl ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2023-11-25T05:09:22.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (5090 ms) 2023-11-25T05:09:22.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2023-11-25T05:09:22.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (5022 ms) 2023-11-25T05:09:22.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2023-11-25T05:09:22.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4318 ms) 2023-11-25T05:09:22.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (14430 ms total) 2023-11-25T05:09:22.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2023-11-25T05:09:22.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2023-11-25T05:09:22.421 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [==========] 19 tests from 4 test suites ran. (226705 ms total) 2023-11-25T05:09:22.421 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ PASSED ] 18 tests. 2023-11-25T05:09:22.421 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2023-11-25T05:09:22.421 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-11-25T05:09:26.966 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: Running main() from gmock_main.cc 2023-11-25T05:09:26.967 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2023-11-25T05:09:26.967 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] Global test environment set-up. 2023-11-25T05:09:26.967 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2023-11-25T05:09:26.967 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2023-11-25T05:09:26.967 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (4412 ms) 2023-11-25T05:09:26.967 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2023-11-25T05:09:26.967 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (54097 ms) 2023-11-25T05:09:26.967 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2023-11-25T05:09:26.967 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (25538 ms) 2023-11-25T05:09:26.967 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2023-11-25T05:09:26.967 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (25576 ms) 2023-11-25T05:09:26.967 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2023-11-25T05:09:26.968 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (25009 ms) 2023-11-25T05:09:26.968 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2023-11-25T05:09:26.968 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (15050 ms) 2023-11-25T05:09:26.968 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2023-11-25T05:09:26.968 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3486 ms) 2023-11-25T05:09:26.968 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2023-11-25T05:09:26.968 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (4007 ms) 2023-11-25T05:09:26.968 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2023-11-25T05:09:26.968 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (4075 ms) 2023-11-25T05:09:26.968 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2023-11-25T05:09:26.968 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3134 ms) 2023-11-25T05:09:26.968 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2023-11-25T05:09:26.968 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2981 ms) 2023-11-25T05:09:26.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2023-11-25T05:09:26.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3952 ms) 2023-11-25T05:09:26.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2023-11-25T05:09:26.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3051 ms) 2023-11-25T05:09:26.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2023-11-25T05:09:26.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (5526 ms) 2023-11-25T05:09:26.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2023-11-25T05:09:26.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (5165 ms) 2023-11-25T05:09:26.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2023-11-25T05:09:26.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3678 ms) 2023-11-25T05:09:26.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2023-11-25T05:09:26.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (4880 ms) 2023-11-25T05:09:26.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2023-11-25T05:09:26.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (4148 ms) 2023-11-25T05:09:26.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2023-11-25T05:09:26.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3947 ms) 2023-11-25T05:09:26.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2023-11-25T05:09:26.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3501 ms) 2023-11-25T05:09:26.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2023-11-25T05:09:26.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3348 ms) 2023-11-25T05:09:26.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2023-11-25T05:09:26.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (4716 ms) 2023-11-25T05:09:26.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2023-11-25T05:09:26.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3553 ms) 2023-11-25T05:09:26.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2023-11-25T05:09:26.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (4404 ms) 2023-11-25T05:09:26.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2023-11-25T05:09:26.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (4087 ms) 2023-11-25T05:09:26.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2023-11-25T05:09:26.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (2534 ms) 2023-11-25T05:09:26.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2023-11-25T05:09:36.261 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:18.459 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:36.268 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:18.466 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:36.328 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:06:18.471 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:36.330 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:06:18.474 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:36.335 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:18.533 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:36.338 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:18.536 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:36.344 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:18.541 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:36.404 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:06:18.548 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:36.411 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:18.609 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:09:54.484 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadolist: : entry=1 expected=1 2023-11-25T05:09:54.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:f905c69b:::2:head 2023-11-25T05:09:54.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:f905c69b:::2:head expected=13:f905c69b:::2:head 2023-11-25T05:09:54.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:f905c69b:::2:head -> 2 2023-11-25T05:09:54.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=2 expected=2 2023-11-25T05:09:54.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:e960b815:::9:head 2023-11-25T05:09:54.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:e960b815:::9:head expected=13:e960b815:::9:head 2023-11-25T05:09:54.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:e960b815:::9:head -> 9 2023-11-25T05:09:54.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=9 expected=9 2023-11-25T05:09:54.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2023-11-25T05:09:54.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:bd63b0f1:::8:head expected=13:bd63b0f1:::8:head 2023-11-25T05:09:54.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2023-11-25T05:09:54.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=8 expected=8 2023-11-25T05:09:54.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:b29083e3:::5:head 2023-11-25T05:09:54.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:b29083e3:::5:head expected=13:b29083e3:::5:head 2023-11-25T05:09:54.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:b29083e3:::5:head -> 5 2023-11-25T05:09:54.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=5 expected=5 2023-11-25T05:09:54.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:89d3ae78:::11:head 2023-11-25T05:09:54.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:89d3ae78:::11:head expected=13:89d3ae78:::11:head 2023-11-25T05:09:54.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2023-11-25T05:09:54.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=11 expected=11 2023-11-25T05:09:54.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:863748b0:::15:head 2023-11-25T05:09:54.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:863748b0:::15:head expected=13:863748b0:::15:head 2023-11-25T05:09:54.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:863748b0:::15:head -> 15 2023-11-25T05:09:54.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=15 expected=15 2023-11-25T05:09:54.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:6cac518f:::0:head 2023-11-25T05:09:54.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:6cac518f:::0:head expected=13:6cac518f:::0:head 2023-11-25T05:09:54.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:6cac518f:::0:head -> 0 2023-11-25T05:09:54.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=0 expected=0 2023-11-25T05:09:54.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 13:62a1935d:::14:head 2023-11-25T05:09:54.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=13:62a1935d:::14:head expected=13:62a1935d:::14:head 2023-11-25T05:09:54.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 13:62a1935d:::14:head -> 14 2023-11-25T05:09:54.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=14 expected=14 2023-11-25T05:09:54.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (11459 ms) 2023-11-25T05:09:54.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2023-11-25T05:09:54.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (127538 ms) 2023-11-25T05:09:54.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2023-11-25T05:09:54.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: split 0/5 -> MIN 13:33333333::::head 2023-11-25T05:09:54.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: split 1/5 -> 13:33333333::::head 13:66666666::::head 2023-11-25T05:09:54.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: split 2/5 -> 13:66666666::::head 13:99999999::::head 2023-11-25T05:09:54.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: split 3/5 -> 13:99999999::::head 13:cccccccc::::head 2023-11-25T05:09:54.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: split 4/5 -> 13:cccccccc::::head MAX 2023-11-25T05:09:54.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (18463 ms) 2023-11-25T05:09:54.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] 7 tests from LibRadosList (247933 ms total) 2023-11-25T05:09:54.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 2023-11-25T05:09:54.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] 3 tests from LibRadosListEC 2023-11-25T05:09:54.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2023-11-25T05:09:54.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1672 ms) 2023-11-25T05:09:54.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2023-11-25T05:09:54.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset foo1,foo2,foo3 2023-11-25T05:09:54.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo1 2023-11-25T05:09:54.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo2 2023-11-25T05:09:54.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo3 2023-11-25T05:09:54.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset foo1,foo4,foo5 2023-11-25T05:09:54.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo4 2023-11-25T05:09:54.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo5 2023-11-25T05:09:54.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo1 2023-11-25T05:09:54.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset foo6,foo7 2023-11-25T05:09:54.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo7 2023-11-25T05:09:54.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo6 2023-11-25T05:09:54.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-11-25T05:09:54.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns1:foo4 2023-11-25T05:09:54.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns1:foo5 2023-11-25T05:09:54.491 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns2:foo7 2023-11-25T05:09:54.491 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns2:foo6 2023-11-25T05:09:54.491 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns1:foo1 2023-11-25T05:09:54.491 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: :foo1 2023-11-25T05:09:54.491 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: :foo2 2023-11-25T05:09:54.491 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: :foo3 2023-11-25T05:09:54.491 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (2278 ms) 2023-11-25T05:10:01.385 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:43.583 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:01.401 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:43.598 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:01.428 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:43.626 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:01.450 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:06:43.594 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:01.456 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:43.654 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:01.475 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:06:43.619 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:01.522 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:43.720 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:01.553 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:43.751 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:01.562 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:06:43.705 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:04.572 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2023-11-25T05:10:04.572 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 1 0 2023-11-25T05:10:04.572 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 10 0 2023-11-25T05:10:04.572 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 13 0 2023-11-25T05:10:04.572 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 7 0 2023-11-25T05:10:04.572 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 14 0 2023-11-25T05:10:04.573 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 0 0 2023-11-25T05:10:04.573 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 15 0 2023-11-25T05:10:04.573 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 11 0 2023-11-25T05:10:04.573 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 5 0 2023-11-25T05:10:04.573 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 8 0 2023-11-25T05:10:04.573 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 6 0 2023-11-25T05:10:04.573 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 3 0 2023-11-25T05:10:04.573 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 4 0 2023-11-25T05:10:04.573 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 12 0 2023-11-25T05:10:04.573 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 9 0 2023-11-25T05:10:04.573 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 2 0 2023-11-25T05:10:04.573 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-11-25T05:10:04.574 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (3950 ms) 2023-11-25T05:10:04.574 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] 3 tests from LibRadosListEC (7901 ms total) 2023-11-25T05:10:04.574 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 2023-11-25T05:10:04.574 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] 1 test from LibRadosListNP 2023-11-25T05:10:04.574 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2023-11-25T05:10:04.574 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3416 ms) 2023-11-25T05:10:04.574 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] 1 test from LibRadosListNP (3416 ms total) 2023-11-25T05:10:04.574 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 2023-11-25T05:10:04.574 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] Global test environment tear-down 2023-11-25T05:10:04.574 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [==========] 11 tests from 3 test suites ran. (268955 ms total) 2023-11-25T05:10:04.574 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ PASSED ] 11 tests. 2023-11-25T05:10:05.481 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2023-11-25T05:10:05.482 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] Global test environment set-up. 2023-11-25T05:10:05.482 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2023-11-25T05:10:05.482 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2023-11-25T05:10:05.482 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2023-11-25T05:10:05.482 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-11-25T05:10:05.482 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2023-11-25T05:10:05.482 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2023-11-25T05:10:05.482 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: seed 92893 2023-11-25T05:10:05.482 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2023-11-25T05:10:05.483 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (13 ms) 2023-11-25T05:10:05.483 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2023-11-25T05:10:05.483 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (20666 ms) 2023-11-25T05:10:05.483 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2023-11-25T05:10:05.483 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (21750 ms) 2023-11-25T05:10:05.483 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2023-11-25T05:10:05.483 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (22798 ms) 2023-11-25T05:10:05.483 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2023-11-25T05:10:05.483 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (7749 ms) 2023-11-25T05:10:05.483 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2023-11-25T05:10:05.483 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (1162 ms) 2023-11-25T05:10:05.483 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2023-11-25T05:10:05.484 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (1858 ms) 2023-11-25T05:10:05.484 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2023-11-25T05:10:05.484 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (168 ms) 2023-11-25T05:10:05.484 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2023-11-25T05:10:05.484 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (68 ms) 2023-11-25T05:10:05.484 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2023-11-25T05:10:05.484 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (341 ms) 2023-11-25T05:10:05.484 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2023-11-25T05:10:05.484 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (50 ms) 2023-11-25T05:10:05.484 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2023-11-25T05:10:05.484 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (177 ms) 2023-11-25T05:10:05.484 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2023-11-25T05:10:05.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (298 ms) 2023-11-25T05:10:05.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2023-11-25T05:10:05.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: osd_max_attr_size = 0 2023-11-25T05:10:05.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2023-11-25T05:10:05.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (88786 ms) 2023-11-25T05:10:05.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2023-11-25T05:10:05.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (13099 ms) 2023-11-25T05:10:05.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2023-11-25T05:10:05.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: waiting for initial deep scrubs... 2023-11-25T05:10:05.485 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: done waiting, doing copies 2023-11-25T05:10:05.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: waiting for final deep scrubs... 2023-11-25T05:10:05.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: done waiting 2023-11-25T05:10:05.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (80698 ms) 2023-11-25T05:10:05.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2023-11-25T05:10:05.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (107 ms) 2023-11-25T05:10:05.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2023-11-25T05:10:05.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (44 ms) 2023-11-25T05:10:05.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2023-11-25T05:10:05.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5907 ms) 2023-11-25T05:10:05.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2023-11-25T05:10:05.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2023-11-25T05:10:05.486 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2023-11-25T05:10:05.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2023-11-25T05:10:05.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2023-11-25T05:10:05.487 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2023-11-25T05:10:17.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [-------- api_c_write_operations: Running main() from gmock_main.cc 2023-11-25T05:10:17.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2023-11-25T05:10:17.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] Global test environment set-up. 2023-11-25T05:10:17.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2023-11-25T05:10:17.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2023-11-25T05:10:17.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2023-11-25T05:10:17.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2023-11-25T05:10:17.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: 2023-11-25T05:10:17.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2023-11-25T05:10:17.894 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2023-11-25T05:10:17.894 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (29014 ms) 2023-11-25T05:10:17.894 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2023-11-25T05:10:17.894 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (8056 ms) 2023-11-25T05:10:17.894 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2023-11-25T05:10:17.894 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (31656 ms) 2023-11-25T05:10:17.894 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2023-11-25T05:10:17.895 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (15104 ms) 2023-11-25T05:10:17.895 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2023-11-25T05:10:17.895 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (28542 ms) 2023-11-25T05:10:17.895 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2023-11-25T05:10:17.895 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (28574 ms) 2023-11-25T05:10:17.896 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2023-11-25T05:10:17.896 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (141114 ms) 2023-11-25T05:10:17.896 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (282060 ms total) 2023-11-25T05:10:17.896 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: 2023-11-25T05:10:17.896 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] Global test environment tear-down 2023-11-25T05:10:17.896 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (282060 ms total) 2023-11-25T05:10:17.897 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2023-11-25T05:10:21.799 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:03.997 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:21.809 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:04.007 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:21.868 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:04.011 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:21.874 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:04.018 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:21.878 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:04.076 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:21.895 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:04.093 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:21.945 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:04.088 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:21.957 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:04.153 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:21.974 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:04.172 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:39.167 INFO:tasks.workunit.client.0.smithi007.stdout:--] 22 tests from LibRadosMiscPP (265740 ms total) 2023-11-25T05:10:39.167 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2023-11-25T05:10:39.167 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2023-11-25T05:10:39.167 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2023-11-25T05:10:39.167 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (5247 ms) 2023-11-25T05:10:39.167 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (5247 ms total) 2023-11-25T05:10:39.167 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2023-11-25T05:10:39.168 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2023-11-25T05:10:39.168 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2023-11-25T05:10:39.168 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (435 ms) 2023-11-25T05:10:39.168 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2023-11-25T05:10:39.168 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (60 ms) 2023-11-25T05:10:39.168 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (495 ms total) 2023-11-25T05:10:39.168 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2023-11-25T05:10:39.168 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2023-11-25T05:10:39.168 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2023-11-25T05:10:39.168 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (467 ms) 2023-11-25T05:10:39.168 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2023-11-25T05:10:39.168 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (85 ms) 2023-11-25T05:10:39.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (552 ms total) 2023-11-25T05:10:39.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2023-11-25T05:10:39.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2023-11-25T05:10:39.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2023-11-25T05:10:39.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (660 ms) 2023-11-25T05:10:39.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2023-11-25T05:10:39.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (334 ms) 2023-11-25T05:10:39.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (994 ms total) 2023-11-25T05:10:39.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2023-11-25T05:10:39.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2023-11-25T05:10:39.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2023-11-25T05:10:39.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1689 ms) 2023-11-25T05:10:39.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1689 ms total) 2023-11-25T05:10:39.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2023-11-25T05:10:39.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] Global test environment tear-down 2023-11-25T05:10:39.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (303509 ms total) 2023-11-25T05:10:39.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ PASSED ] 31 tests. 2023-11-25T05:10:41.835 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:24.033 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:41.894 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:24.038 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:41.894 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:24.092 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:41.900 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:24.044 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:41.955 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:24.099 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:41.956 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:24.154 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:41.968 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:24.166 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:41.998 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:24.196 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:42.033 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:24.231 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:49.425 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:31.623 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:49.436 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:31.634 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:49.480 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:31.678 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:49.506 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:31.650 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:49.514 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:31.658 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:49.519 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:31.717 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:49.524 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:31.722 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:49.542 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:31.686 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:10:49.554 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:31.752 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:01.705 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:43.903 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:01.710 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:43.909 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:01.740 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:43.938 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:01.757 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:43.901 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:01.758 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:43.956 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:01.759 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:43.903 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:01.761 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:43.959 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:01.787 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:43.931 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:01.790 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:43.988 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:17.011 INFO:tasks.workunit.client.0.smithi007.stdout:eWrite 2023-11-25T05:11:17.011 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6856 ms) 2023-11-25T05:11:17.011 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2023-11-25T05:11:17.011 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (8737 ms) 2023-11-25T05:11:17.012 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2023-11-25T05:11:17.012 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7220 ms) 2023-11-25T05:11:17.012 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2023-11-25T05:11:17.012 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7544 ms) 2023-11-25T05:11:17.012 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2023-11-25T05:11:17.012 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7783 ms) 2023-11-25T05:11:17.012 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2023-11-25T05:11:17.012 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7370 ms) 2023-11-25T05:11:17.012 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2023-11-25T05:11:17.013 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (8944 ms) 2023-11-25T05:11:17.013 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2023-11-25T05:11:17.013 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6948 ms) 2023-11-25T05:11:17.013 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2023-11-25T05:11:17.013 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (8025 ms) 2023-11-25T05:11:17.013 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2023-11-25T05:11:17.013 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7602 ms) 2023-11-25T05:11:17.013 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2023-11-25T05:11:17.013 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7693 ms) 2023-11-25T05:11:17.013 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2023-11-25T05:11:17.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7092 ms) 2023-11-25T05:11:17.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2023-11-25T05:11:17.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7648 ms) 2023-11-25T05:11:17.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2023-11-25T05:11:17.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7089 ms) 2023-11-25T05:11:17.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2023-11-25T05:11:17.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7049 ms) 2023-11-25T05:11:17.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2023-11-25T05:11:17.014 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (8951 ms) 2023-11-25T05:11:17.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (122551 ms total) 2023-11-25T05:11:17.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: 2023-11-25T05:11:17.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] Global test environment tear-down 2023-11-25T05:11:17.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (341470 ms total) 2023-11-25T05:11:17.015 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ PASSED ] 42 tests. 2023-11-25T05:11:21.909 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:04.107 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:21.977 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:04.175 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:22.032 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:04.230 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:22.032 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:04.177 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:22.043 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:04.187 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:22.049 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:04.247 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:22.088 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:04.286 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:22.111 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:04.255 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:22.141 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:04.339 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:42.097 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:24.294 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:42.104 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:24.302 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:42.163 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:24.361 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:42.170 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:24.314 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:42.175 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:24.373 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:42.180 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:24.324 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:42.205 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:24.403 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:42.234 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:24.378 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:11:42.265 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:24.463 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:02.037 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:44.235 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:02.042 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:44.240 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:02.085 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:44.229 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:02.090 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:44.235 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:02.093 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:44.291 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:02.095 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:44.293 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:02.097 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:44.296 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:02.126 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:44.270 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:02.134 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:44.332 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:11.505 INFO:tasks.workunit.client.0.smithi007.stdout:sAio.RoundTripAppendPP (3563 ms) 2023-11-25T05:12:11.505 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2023-11-25T05:12:11.505 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3069 ms) 2023-11-25T05:12:11.505 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2023-11-25T05:12:11.505 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3472 ms) 2023-11-25T05:12:11.505 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2023-11-25T05:12:11.505 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3762 ms) 2023-11-25T05:12:11.505 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2023-11-25T05:12:11.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: setting pool EIO 2023-11-25T05:12:11.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: waiting for 321 2023-11-25T05:12:11.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: max_success 0, min_failed 1 2023-11-25T05:12:11.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (8873 ms) 2023-11-25T05:12:11.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2023-11-25T05:12:11.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4297 ms) 2023-11-25T05:12:11.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (254895 ms total) 2023-11-25T05:12:11.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: 2023-11-25T05:12:11.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2023-11-25T05:12:11.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2023-11-25T05:12:11.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3459 ms) 2023-11-25T05:12:11.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2023-11-25T05:12:11.506 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (10610 ms) 2023-11-25T05:12:11.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2023-11-25T05:12:11.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (17145 ms) 2023-11-25T05:12:11.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2023-11-25T05:12:11.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3429 ms) 2023-11-25T05:12:11.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (34643 ms total) 2023-11-25T05:12:11.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: 2023-11-25T05:12:11.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2023-11-25T05:12:11.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-11-25T05:12:11.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (4377 ms) 2023-11-25T05:12:11.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (4377 ms total) 2023-11-25T05:12:11.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: 2023-11-25T05:12:11.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2023-11-25T05:12:11.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2023-11-25T05:12:11.507 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13508 ms) 2023-11-25T05:12:11.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2023-11-25T05:12:11.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7573 ms) 2023-11-25T05:12:11.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2023-11-25T05:12:11.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7219 ms) 2023-11-25T05:12:11.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2023-11-25T05:12:11.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7420 ms) 2023-11-25T05:12:11.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2023-11-25T05:12:11.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (2800 ms) 2023-11-25T05:12:11.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2023-11-25T05:12:11.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (9032 ms) 2023-11-25T05:12:11.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2023-11-25T05:12:11.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7966 ms) 2023-11-25T05:12:11.508 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2023-11-25T05:12:11.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7178 ms) 2023-11-25T05:12:11.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2023-11-25T05:12:11.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7852 ms) 2023-11-25T05:12:11.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2023-11-25T05:12:11.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7532 ms) 2023-11-25T05:12:11.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2023-11-25T05:12:11.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7980 ms) 2023-11-25T05:12:11.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2023-11-25T05:12:11.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (8066 ms) 2023-11-25T05:12:11.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2023-11-25T05:12:11.509 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7918 ms) 2023-11-25T05:12:22.125 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:04.323 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:22.128 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:04.326 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:22.166 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:04.365 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:22.170 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:04.314 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:22.171 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:04.315 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:22.174 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:04.372 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:22.177 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:04.375 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:22.208 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:04.352 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:22.224 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:04.422 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:38.319 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:20.517 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:38.385 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:20.583 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:38.389 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:20.533 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:38.415 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:20.559 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:38.440 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:20.638 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:38.447 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:20.645 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:38.452 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:20.650 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:38.463 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:20.606 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:38.516 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:20.714 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:42.197 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:24.394 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:42.199 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:24.397 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:42.211 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:24.409 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:42.252 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:24.450 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:42.253 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:24.451 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:42.257 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:24.400 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:42.257 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:24.402 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:42.267 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:24.465 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:42.270 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:24.414 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:12:58.444 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2023-11-25T05:12:58.444 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3304 ms) 2023-11-25T05:12:58.444 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2023-11-25T05:12:58.444 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6712 ms) 2023-11-25T05:12:58.444 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2023-11-25T05:12:58.444 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6981 ms) 2023-11-25T05:12:58.444 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2023-11-25T05:12:58.444 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7545 ms) 2023-11-25T05:12:58.445 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2023-11-25T05:12:58.445 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6560 ms) 2023-11-25T05:12:58.445 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2023-11-25T05:12:58.445 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7968 ms) 2023-11-25T05:12:58.445 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2023-11-25T05:12:58.445 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7866 ms) 2023-11-25T05:12:58.445 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (148980 ms total) 2023-11-25T05:12:58.445 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: 2023-11-25T05:12:58.445 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] Global test environment tear-down 2023-11-25T05:12:58.445 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (442896 ms total) 2023-11-25T05:12:58.445 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ PASSED ] 57 tests. 2023-11-25T05:13:02.327 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:44.525 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:02.330 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:44.528 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:02.361 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:44.559 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:02.380 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:44.578 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:02.381 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:44.579 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:02.390 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:44.534 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:02.391 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:44.535 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:02.421 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:44.565 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:02.425 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:44.623 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:22.415 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:04.613 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:22.418 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:04.616 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:22.456 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:04.654 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:22.473 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:04.617 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:22.479 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:04.623 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:22.481 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:04.680 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:22.485 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:04.683 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:22.522 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:04.666 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:22.527 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:04.725 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:42.435 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:24.633 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:42.492 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:24.690 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:42.531 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:24.730 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:42.536 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:24.680 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:42.538 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:24.682 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:42.541 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:24.739 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:42.544 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:24.742 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:42.570 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:24.714 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:13:42.578 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:24.776 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:02.544 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:44.741 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:02.550 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:44.748 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:02.584 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:44.782 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:02.591 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:44.735 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:02.594 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:44.738 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:02.618 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:44.816 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:02.621 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:44.820 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:02.626 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:44.770 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:02.675 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:44.873 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:07.377 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2023-11-25T05:14:07.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] Global test environment set-up. 2023-11-25T05:14:07.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2023-11-25T05:14:07.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: seed 92908 2023-11-25T05:14:07.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2023-11-25T05:14:07.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (14612 ms) 2023-11-25T05:14:07.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2023-11-25T05:14:07.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (22993 ms) 2023-11-25T05:14:07.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2023-11-25T05:14:07.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (672 ms) 2023-11-25T05:14:07.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (38277 ms total) 2023-11-25T05:14:07.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: 2023-11-25T05:14:07.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2023-11-25T05:14:07.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2023-11-25T05:14:07.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (62550 ms) 2023-11-25T05:14:07.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2023-11-25T05:14:07.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (54092 ms) 2023-11-25T05:14:07.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2023-11-25T05:14:07.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (32187 ms) 2023-11-25T05:14:07.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2023-11-25T05:14:07.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: my_snaps [4] 2023-11-25T05:14:07.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: my_snaps [5,4] 2023-11-25T05:14:07.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: my_snaps [6,5,4] 2023-11-25T05:14:07.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: my_snaps [7,6,5,4] 2023-11-25T05:14:07.381 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: promoting some heads 2023-11-25T05:14:07.381 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: promoting from clones for snap 7 2023-11-25T05:14:07.381 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: promoting from clones for snap 6 2023-11-25T05:14:07.381 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: promoting from clones for snap 5 2023-11-25T05:14:07.381 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: promoting from clones for snap 4 2023-11-25T05:14:07.381 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: waiting for scrubs... 2023-11-25T05:14:07.381 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: done waiting 2023-11-25T05:14:07.381 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (153381 ms) 2023-11-25T05:14:07.382 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2023-11-25T05:14:07.382 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9842 ms) 2023-11-25T05:14:07.382 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2023-11-25T05:14:07.382 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8833 ms) 2023-11-25T05:14:07.382 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2023-11-25T05:14:07.382 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (9603 ms) 2023-11-25T05:14:07.382 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2023-11-25T05:14:07.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (10231 ms) 2023-11-25T05:14:07.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2023-11-25T05:14:07.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (11585 ms) 2023-11-25T05:14:07.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2023-11-25T05:14:07.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (10004 ms) 2023-11-25T05:14:07.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2023-11-25T05:14:07.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10137 ms) 2023-11-25T05:14:07.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2023-11-25T05:14:07.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (40647 ms) 2023-11-25T05:14:07.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2023-11-25T05:14:07.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (9970 ms) 2023-11-25T05:14:07.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2023-11-25T05:14:07.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (9845 ms) 2023-11-25T05:14:07.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2023-11-25T05:14:07.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (14606 ms) 2023-11-25T05:14:07.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2023-11-25T05:14:07.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (9193 ms) 2023-11-25T05:14:07.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2023-11-25T05:14:07.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (11464 ms) 2023-11-25T05:14:22.694 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:04.892 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:22.699 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:04.897 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:22.701 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:04.845 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:22.741 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:04.939 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:22.744 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:04.942 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:22.747 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:04.891 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:22.780 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:04.978 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:22.781 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:04.980 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:22.785 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:04.929 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:28.875 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2023-11-25T05:14:28.875 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: hmm, no HitSet yet 2023-11-25T05:14:28.875 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: ok, hit_set contains 200:602f83fe:::foo:head 2023-11-25T05:14:28.875 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8937 ms) 2023-11-25T05:14:28.875 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2023-11-25T05:14:28.875 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg_num = 32 2023-11-25T05:14:28.875 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 0 ls 1700889263,0 2023-11-25T05:14:28.876 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 1 ls 1700889264,0 2023-11-25T05:14:28.876 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 2 ls 1700889263,0 2023-11-25T05:14:28.876 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 3 ls 1700889263,0 2023-11-25T05:14:28.876 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 4 ls 1700889264,0 2023-11-25T05:14:28.876 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 5 ls 1700889264,0 2023-11-25T05:14:28.876 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 6 ls 1700889263,0 2023-11-25T05:14:28.876 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 7 ls 1700889264,0 2023-11-25T05:14:28.876 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 8 ls 1700889263,0 2023-11-25T05:14:28.876 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 9 ls 1700889263,0 2023-11-25T05:14:28.876 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 10 ls 1700889263,0 2023-11-25T05:14:28.876 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 11 ls 1700889264,0 2023-11-25T05:14:28.876 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 12 ls 1700889263,0 2023-11-25T05:14:28.877 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 13 ls 1700889264,0 2023-11-25T05:14:28.877 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 14 ls 1700889263,0 2023-11-25T05:14:28.877 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 15 ls 1700889264,0 2023-11-25T05:14:28.877 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 16 ls 1700889263,0 2023-11-25T05:14:28.877 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 17 ls 1700889264,0 2023-11-25T05:14:28.877 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 18 ls 1700889263,0 2023-11-25T05:14:28.877 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 19 ls 1700889263,0 2023-11-25T05:14:28.877 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 20 ls 1700889264,0 2023-11-25T05:14:28.877 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 21 ls 1700889263,0 2023-11-25T05:14:28.877 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 22 ls 1700889263,0 2023-11-25T05:14:28.877 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 23 ls 1700889263,0 2023-11-25T05:14:28.877 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 24 ls 1700889263,0 2023-11-25T05:14:28.877 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 25 ls 1700889263,0 2023-11-25T05:14:28.878 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 26 ls 1700889264,0 2023-11-25T05:14:28.878 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 27 ls 1700889263,0 2023-11-25T05:14:28.878 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 28 ls 1700889264,0 2023-11-25T05:14:28.878 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 29 ls 1700889263,0 2023-11-25T05:14:28.878 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 30 ls 1700889263,0 2023-11-25T05:14:28.878 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 31 ls 1700889263,0 2023-11-25T05:14:28.878 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg_num = 32 2023-11-25T05:14:28.878 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:6cac518f:::0:head 2023-11-25T05:14:28.878 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:02547ec2:::1:head 2023-11-25T05:14:28.878 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:f905c69b:::2:head 2023-11-25T05:14:28.878 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:cfc208b3:::3:head 2023-11-25T05:14:28.878 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:d83876eb:::4:head 2023-11-25T05:14:28.879 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:b29083e3:::5:head 2023-11-25T05:14:28.879 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:c4fdafeb:::6:head 2023-11-25T05:14:28.879 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:5c6b0b28:::7:head 2023-11-25T05:14:28.879 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:bd63b0f1:::8:head 2023-11-25T05:14:28.879 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:e960b815:::9:head 2023-11-25T05:14:28.879 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:52ea6a34:::10:head 2023-11-25T05:14:28.879 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:89d3ae78:::11:head 2023-11-25T05:14:28.879 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:de5d7c5f:::12:head 2023-11-25T05:14:28.880 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:566253c9:::13:head 2023-11-25T05:14:28.880 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:62a1935d:::14:head 2023-11-25T05:14:28.880 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:863748b0:::15:head 2023-11-25T05:14:28.880 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:3958e169:::16:head 2023-11-25T05:14:28.880 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:4d4dabf9:::17:head 2023-11-25T05:14:28.880 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:8391935d:::18:head 2023-11-25T05:14:28.880 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:28883081:::19:head 2023-11-25T05:14:28.880 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:69259c59:::20:head 2023-11-25T05:14:28.880 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:4bdb80b7:::21:head 2023-11-25T05:14:28.880 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:a11c5d71:::22:head 2023-11-25T05:14:28.880 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:271af37b:::23:head 2023-11-25T05:14:28.880 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:95b121be:::24:head 2023-11-25T05:14:28.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:58d1031b:::25:head 2023-11-25T05:14:28.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:0a050783:::26:head 2023-11-25T05:14:28.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:c709704c:::27:head 2023-11-25T05:14:28.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:cbe56eaf:::28:head 2023-11-25T05:14:28.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:86b4b162:::29:head 2023-11-25T05:14:28.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:70d89383:::30:head 2023-11-25T05:14:28.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:dd450c7c:::31:head 2023-11-25T05:14:28.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:6d5729b1:::32:head 2023-11-25T05:14:28.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:c388f3fb:::33:head 2023-11-25T05:14:28.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:56cfea31:::34:head 2023-11-25T05:14:28.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:9dbc1bf7:::35:head 2023-11-25T05:14:28.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:40b74ccd:::36:head 2023-11-25T05:14:28.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:4d5aaf42:::37:head 2023-11-25T05:14:28.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:920f362c:::38:head 2023-11-25T05:14:28.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:6cc53222:::39:head 2023-11-25T05:14:28.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:9cad833f:::40:head 2023-11-25T05:14:28.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:1ea84d41:::41:head 2023-11-25T05:14:28.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:c4480ef6:::42:head 2023-11-25T05:14:28.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:a694361e:::43:head 2023-11-25T05:14:28.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:d1bd33e9:::44:head 2023-11-25T05:14:28.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:ddc2cd5d:::45:head 2023-11-25T05:14:28.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:2b782207:::46:head 2023-11-25T05:14:28.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:7b187fca:::47:head 2023-11-25T05:14:28.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:90ecdf6f:::48:head 2023-11-25T05:14:28.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:a5ed95fe:::49:head 2023-11-25T05:14:28.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:ea0eaa55:::50:head 2023-11-25T05:14:28.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:f33ef17b:::51:head 2023-11-25T05:14:28.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:a0d1b2f6:::52:head 2023-11-25T05:14:28.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:60c5229e:::53:head 2023-11-25T05:14:28.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:edcbc575:::54:head 2023-11-25T05:14:28.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:102cf253:::55:head 2023-11-25T05:14:28.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:efb7fb0b:::56:head 2023-11-25T05:14:28.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:50d0a326:::57:head 2023-11-25T05:14:28.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:d4dc5daf:::58:head 2023-11-25T05:14:28.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:3a130462:::59:head 2023-11-25T05:14:28.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:ec87ed71:::60:head 2023-11-25T05:14:28.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:d5bc9454:::61:head 2023-11-25T05:14:28.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:3ddfe313:::62:head 2023-11-25T05:14:28.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:7c2816b9:::63:head 2023-11-25T05:14:28.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:47e00e4d:::64:head 2023-11-25T05:14:28.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:c6410c18:::65:head 2023-11-25T05:14:28.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:b48ed237:::66:head 2023-11-25T05:14:28.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:cd63ad31:::67:head 2023-11-25T05:14:28.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:b179e92b:::68:head 2023-11-25T05:14:28.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:0d9f741a:::69:head 2023-11-25T05:14:28.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:6d3352ae:::70:head 2023-11-25T05:14:28.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:c6d5c19e:::71:head 2023-11-25T05:14:28.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:bc4729c3:::72:head 2023-11-25T05:14:28.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:77e930b9:::73:head 2023-11-25T05:14:28.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:0abeecfd:::74:head 2023-11-25T05:14:28.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:b7c37e15:::75:head 2023-11-25T05:14:28.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:b6378398:::76:head 2023-11-25T05:14:28.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:02bd68de:::77:head 2023-11-25T05:14:28.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:cc795d2d:::78:head 2023-11-25T05:14:28.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:630d4fea:::79:head 2023-11-25T05:14:28.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:e0d29ef5:::80:head 2023-11-25T05:14:28.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:fd6f13d2:::81:head 2023-11-25T05:14:28.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:606461d5:::82:head 2023-11-25T05:14:28.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:eadbdc43:::83:head 2023-11-25T05:14:28.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:8761d0bb:::84:head 2023-11-25T05:14:28.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:9ef0186f:::85:head 2023-11-25T05:14:28.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:e0d41294:::86:head 2023-11-25T05:14:28.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:961de695:::87:head 2023-11-25T05:14:28.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:1423148f:::88:head 2023-11-25T05:14:28.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:633a8fa2:::89:head 2023-11-25T05:14:28.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:a8653809:::90:head 2023-11-25T05:14:28.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:3dac8b33:::91:head 2023-11-25T05:14:28.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:35aad435:::92:head 2023-11-25T05:14:28.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:f6dcc343:::93:head 2023-11-25T05:14:28.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:dbbdad87:::94:head 2023-11-25T05:14:28.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:1cb48ce0:::95:head 2023-11-25T05:14:28.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:03cd461c:::96:head 2023-11-25T05:14:28.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:17a4ea99:::97:head 2023-11-25T05:14:28.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:9993c9a7:::98:head 2023-11-25T05:14:28.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:6394211c:::99:head 2023-11-25T05:14:28.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:94c7ae57:::100:head 2023-11-25T05:14:28.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:6fdee5bb:::101:head 2023-11-25T05:14:28.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:9a477fd1:::102:head 2023-11-25T05:14:28.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:eb850916:::103:head 2023-11-25T05:14:28.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:affc56b9:::104:head 2023-11-25T05:14:28.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:b42dc814:::105:head 2023-11-25T05:14:28.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:f319f8f0:::106:head 2023-11-25T05:14:28.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:9a40b9de:::107:head 2023-11-25T05:14:28.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:8b524f28:::108:head 2023-11-25T05:14:28.888 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:e3de589f:::109:head 2023-11-25T05:14:28.888 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:90f90a5b:::110:head 2023-11-25T05:14:28.888 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:a7b4f1d7:::111:head 2023-11-25T05:14:28.888 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:af51766e:::112:head 2023-11-25T05:14:28.888 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:b6f90bd1:::113:head 2023-11-25T05:14:28.888 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:e0261208:::114:head 2023-11-25T05:14:28.888 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:c9569ef7:::115:head 2023-11-25T05:14:28.888 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:61bebe50:::116:head 2023-11-25T05:14:28.888 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:fe93412b:::117:head 2023-11-25T05:14:28.888 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:d3d38bee:::118:head 2023-11-25T05:14:28.888 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:3100ba0c:::119:head 2023-11-25T05:14:28.888 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:d0560ada:::120:head 2023-11-25T05:14:28.889 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:f0ea8b35:::121:head 2023-11-25T05:14:28.889 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:766f231a:::122:head 2023-11-25T05:14:28.889 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:a07a2582:::123:head 2023-11-25T05:14:28.889 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:bd7c6b3a:::124:head 2023-11-25T05:14:28.889 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:fb2ddaff:::125:head 2023-11-25T05:14:28.889 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:4408e1fe:::126:head 2023-11-25T05:14:28.889 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:ee1df7a7:::127:head 2023-11-25T05:14:28.889 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:c3002909:::128:head 2023-11-25T05:14:28.889 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:4f48ffa9:::129:head 2023-11-25T05:14:28.889 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:edf38733:::130:head 2023-11-25T05:14:28.889 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:c08425c0:::131:head 2023-11-25T05:14:28.889 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:5f902d98:::132:head 2023-11-25T05:14:28.889 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:41ea2c93:::133:head 2023-11-25T05:14:28.890 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:813cee13:::134:head 2023-11-25T05:14:28.890 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:0131818d:::135:head 2023-11-25T05:14:28.890 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:26ba5a85:::136:head 2023-11-25T05:14:28.890 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:381b8a5a:::137:head 2023-11-25T05:14:28.890 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:28797e47:::138:head 2023-11-25T05:14:28.890 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:bfca7f22:::139:head 2023-11-25T05:14:28.890 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:36807075:::140:head 2023-11-25T05:14:28.890 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:80b03975:::141:head 2023-11-25T05:14:28.890 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:5c15709b:::142:head 2023-11-25T05:14:28.890 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:f39ea15e:::143:head 2023-11-25T05:14:28.890 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:ea992956:::144:head 2023-11-25T05:14:28.890 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:48887b1c:::145:head 2023-11-25T05:14:28.891 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:9f24a9dd:::146:head 2023-11-25T05:14:28.891 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:987f100b:::147:head 2023-11-25T05:14:28.891 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:d2dd3581:::148:head 2023-11-25T05:14:28.891 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:7fed1808:::149:head 2023-11-25T05:14:28.891 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:c80b70e9:::150:head 2023-11-25T05:14:28.891 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:85ed90f9:::151:head 2023-11-25T05:14:28.891 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:36428b24:::152:head 2023-11-25T05:14:28.891 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:d044c34a:::153:head 2023-11-25T05:14:28.891 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:7c18bf58:::154:head 2023-11-25T05:14:28.891 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:d1c21232:::155:head 2023-11-25T05:14:28.891 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:a7a3c575:::156:head 2023-11-25T05:14:28.891 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:87da0633:::157:head 2023-11-25T05:14:28.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:d5ac3822:::158:head 2023-11-25T05:14:28.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:3f20522d:::159:head 2023-11-25T05:14:28.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:6ca26563:::160:head 2023-11-25T05:14:28.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:532ce135:::161:head 2023-11-25T05:14:28.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:c78863e6:::162:head 2023-11-25T05:14:36.984 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:19.182 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:36.989 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:19.187 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:37.012 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:19.211 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:37.037 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:19.180 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:37.038 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:19.182 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:37.043 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:19.241 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:37.045 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:19.244 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:37.057 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:19.201 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:37.066 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:19.265 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:42.929 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:25.125 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:42.930 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:25.127 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:42.945 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:25.143 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:42.974 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:25.118 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:42.975 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:25.119 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:42.977 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:25.176 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:42.978 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:25.177 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:42.993 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:25.136 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:14:42.997 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:25.195 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:02.982 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:45.180 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:02.986 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:45.183 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:03.020 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:45.218 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:03.052 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:45.196 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:03.054 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:45.198 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:03.061 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:45.259 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:03.063 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:45.261 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:03.095 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:45.239 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:03.096 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:45.294 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:22.891 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:05.088 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:22.896 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:05.094 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:22.932 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:05.129 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:22.941 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:05.085 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:22.943 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:05.087 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:22.944 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:05.142 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:22.949 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:05.147 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:22.974 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:05.118 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:23.000 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:05.197 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:32.921 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:15.118 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:32.922 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:15.120 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:32.936 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:15.134 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:32.965 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:15.109 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:32.968 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:15.167 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:32.969 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:15.113 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:32.970 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:15.169 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:32.981 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:15.125 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:32.982 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:15.181 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:38.627 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:7462ddf6:::163:head 2023-11-25T05:15:38.627 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:5d165639:::164:head 2023-11-25T05:15:38.627 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:f43765fc:::165:head 2023-11-25T05:15:38.627 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:b4c720e9:::166:head 2023-11-25T05:15:38.627 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:e694b040:::167:head 2023-11-25T05:15:38.627 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:afa38db2:::168:head 2023-11-25T05:15:38.627 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:77ba9f53:::169:head 2023-11-25T05:15:38.627 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:87495034:::170:head 2023-11-25T05:15:38.627 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:7c96bf0e:::171:head 2023-11-25T05:15:38.627 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:dbe346cc:::172:head 2023-11-25T05:15:38.627 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:e943ec24:::173:head 2023-11-25T05:15:38.628 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:f97a9c0c:::174:head 2023-11-25T05:15:38.628 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:6f26e74d:::175:head 2023-11-25T05:15:38.628 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:4f95e106:::176:head 2023-11-25T05:15:38.628 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:0e6f2f8f:::177:head 2023-11-25T05:15:38.628 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:05db05f1:::178:head 2023-11-25T05:15:38.628 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:38a78d66:::179:head 2023-11-25T05:15:38.628 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:d095610b:::180:head 2023-11-25T05:15:38.628 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:a1a9d709:::181:head 2023-11-25T05:15:38.628 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:1e5d39db:::182:head 2023-11-25T05:15:38.628 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:f7df4fb9:::183:head 2023-11-25T05:15:38.628 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:03a7f161:::184:head 2023-11-25T05:15:38.628 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:ba70721e:::185:head 2023-11-25T05:15:38.628 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:28e5662d:::186:head 2023-11-25T05:15:38.629 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:973d52de:::187:head 2023-11-25T05:15:38.629 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:4303eb1c:::188:head 2023-11-25T05:15:38.629 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:b990b48e:::189:head 2023-11-25T05:15:38.629 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:29b8165b:::190:head 2023-11-25T05:15:38.629 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:3547f197:::191:head 2023-11-25T05:15:38.629 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:7e260936:::192:head 2023-11-25T05:15:38.629 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:1abec7b1:::193:head 2023-11-25T05:15:38.629 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:10fdda93:::194:head 2023-11-25T05:15:38.629 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:15817eea:::195:head 2023-11-25T05:15:38.629 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:770bab57:::196:head 2023-11-25T05:15:38.629 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:ed9e13e7:::197:head 2023-11-25T05:15:38.629 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:71471a8f:::198:head 2023-11-25T05:15:38.629 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 201:10fb1d02:::199:head 2023-11-25T05:15:38.630 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (21768 ms) 2023-11-25T05:15:38.630 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2023-11-25T05:15:38.630 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889286,0 2023-11-25T05:15:38.630 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: first is 1700889286 2023-11-25T05:15:38.630 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889286,0 2023-11-25T05:15:38.630 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889286,0 2023-11-25T05:15:38.630 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889286,1700889288,1700889289,0 2023-11-25T05:15:38.630 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889286,1700889288,1700889289,0 2023-11-25T05:15:38.630 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889286,1700889288,1700889289,0 2023-11-25T05:15:38.630 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889286,1700889288,1700889289,1700889291,1700889292,0 2023-11-25T05:15:38.630 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889286,1700889288,1700889289,1700889291,1700889292,0 2023-11-25T05:15:38.630 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889286,1700889288,1700889289,1700889291,1700889292,0 2023-11-25T05:15:38.630 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889286,1700889288,1700889289,1700889291,1700889292,1700889294,1700889295,0 2023-11-25T05:15:38.631 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889286,1700889288,1700889289,1700889291,1700889292,1700889294,1700889295,0 2023-11-25T05:15:38.631 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889286,1700889288,1700889289,1700889291,1700889292,1700889294,1700889295,0 2023-11-25T05:15:38.631 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889289,1700889291,1700889292,1700889294,1700889295,1700889297,1700889298,0 2023-11-25T05:15:38.631 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: first now 1700889289, trimmed 2023-11-25T05:15:38.631 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22936 ms) 2023-11-25T05:15:38.631 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2023-11-25T05:15:38.631 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: foo0 2023-11-25T05:15:38.631 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-11-25T05:15:38.631 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (16158 ms) 2023-11-25T05:15:38.631 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2023-11-25T05:15:43.006 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:25.204 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:43.014 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:25.158 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:43.017 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:25.215 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:43.057 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:25.201 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:43.060 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:25.259 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:43.062 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:25.260 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:43.075 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:25.220 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:43.091 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:25.290 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:43.094 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:25.292 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:48.108 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:30.307 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:48.111 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:30.310 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:48.125 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:30.323 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:48.155 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:30.299 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:48.157 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:30.301 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:48.160 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:30.358 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:48.161 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:30.359 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:48.170 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:30.314 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:15:48.174 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:30.373 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.026 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:45.224 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.029 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:45.227 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.048 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:45.246 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.071 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:45.215 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.073 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:45.217 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.081 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:45.279 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.082 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:45.281 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.091 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:45.235 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.100 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:45.298 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.879 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:46.023 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.880 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:46.024 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.888 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:46.032 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.932 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:46.130 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.933 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:46.131 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.942 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:46.140 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.975 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:46.173 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.976 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:46.174 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:03.985 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:46.183 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:18.329 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:00.527 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:18.338 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:00.533 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:18.339 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:00.537 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:18.341 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:00.485 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:18.341 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:00.539 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:18.344 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:00.488 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:18.444 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:00.588 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:18.460 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:00.658 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:18.475 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:00.673 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:23.156 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:05.354 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:23.157 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:05.355 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:23.180 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:05.378 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:23.201 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:05.345 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:23.204 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:05.348 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:23.212 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:05.410 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:23.215 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:05.413 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:23.217 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:05.361 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:23.237 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:05.435 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:33.230 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:15.428 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:33.233 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:15.431 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:33.255 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:15.453 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:33.278 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:15.422 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:33.281 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:15.425 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:33.282 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:15.480 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:33.286 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:15.484 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:33.300 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:15.444 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:33.308 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:15.506 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:43.403 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:25.600 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:43.404 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:25.602 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:43.426 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:25.624 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:43.448 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:25.592 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:43.449 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:25.593 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:43.462 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:25.606 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:43.463 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:25.662 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:43.467 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:25.665 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:43.488 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:25.686 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:53.440 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:35.638 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:53.444 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:35.643 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:53.463 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:35.661 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:53.494 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:35.638 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:53.495 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:35.693 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:53.495 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:35.639 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:53.496 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:35.694 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:53.514 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:35.658 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:16:53.514 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:35.712 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:03.376 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:45.575 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:03.431 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:45.629 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:03.433 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:45.577 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:03.435 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:45.579 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:03.450 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:45.594 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:03.452 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:45.650 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:03.492 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:45.690 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:03.494 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:45.692 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:03.509 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:45.707 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:13.383 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:55.581 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:13.385 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:55.583 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:13.408 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:55.606 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:13.441 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:55.584 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:13.442 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:55.640 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:13.442 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:55.587 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:13.444 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:55.642 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:13.462 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:13:55.606 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:13.467 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:55.666 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:23.517 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:05.715 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:23.518 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:05.717 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:23.533 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:05.730 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:23.566 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:14:05.710 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:23.567 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:14:05.711 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:23.571 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:05.769 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:23.573 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:05.771 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:23.582 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:14:05.726 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:23.587 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:05.784 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:28.583 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:10.781 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:28.584 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:10.782 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:28.598 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:10.797 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:28.621 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:14:10.765 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:28.628 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:14:10.773 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:28.634 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:10.832 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:28.635 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:10.833 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:28.644 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:14:10.788 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:28.653 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:10.851 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:43.609 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:25.807 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:43.610 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:25.809 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:43.634 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:25.832 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:43.660 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:25.858 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:43.662 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:25.860 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:43.666 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:14:25.810 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:43.669 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:14:25.812 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:43.688 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:25.886 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:17:43.690 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:14:25.834 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:03.802 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:46.000 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:03.804 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:46.003 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:03.835 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:46.033 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:03.852 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:14:45.995 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:03.853 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:14:45.996 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:03.864 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:46.062 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:03.866 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:46.064 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:03.869 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:14:46.013 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:03.885 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:46.083 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:23.843 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:06.041 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:23.848 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:06.047 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:23.878 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:06.076 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:23.909 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:06.053 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:23.911 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:06.055 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:23.912 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:06.109 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:23.912 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:06.110 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:23.933 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:06.077 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:23.936 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:06.134 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:38.835 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:21.033 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:38.837 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:21.035 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:38.865 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:21.062 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:38.887 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:21.085 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:38.893 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:21.091 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:38.900 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:21.044 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:38.903 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:21.047 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:38.919 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:21.117 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:38.926 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:21.070 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:43.749 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:25.947 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:43.751 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:25.949 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:43.765 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:25.963 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:43.798 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:25.942 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:43.802 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:25.945 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:43.802 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:25.999 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:43.803 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:26.001 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:43.814 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:25.957 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:43.815 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:26.013 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:53.798 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:35.994 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:53.798 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:35.996 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:53.812 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:36.009 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:53.851 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:35.994 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:53.852 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:36.050 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:53.853 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:35.997 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:53.853 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:36.052 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:53.865 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:36.009 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:53.867 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:36.065 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:18:54.952 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-25T05:18:54.935+0000 23a7f640 -1 snap_mapper.add_oid found existing snaps mapped on 219:006d31ca:test-rados-api-smithi007-92908-80::foo:2, removing 2023-11-25T05:18:54.971 INFO:tasks.ceph.osd.6.smithi090.stderr:2023-11-25T05:18:54.962+0000 2a28c640 -1 snap_mapper.add_oid found existing snaps mapped on 219:006d31ca:test-rados-api-smithi007-92908-80::foo:2, removing 2023-11-25T05:19:03.842 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:46.040 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:03.843 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:46.042 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:03.865 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:46.063 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:03.883 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:46.027 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:03.885 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:46.029 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:03.889 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:46.087 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:03.892 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:46.090 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:03.907 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:46.051 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:03.914 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:46.113 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:09.036 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:51.234 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:09.040 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:51.238 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:09.064 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:51.262 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:09.065 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:51.263 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:09.104 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:51.247 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:09.108 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:51.306 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:09.110 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:51.254 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:09.131 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:51.275 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:09.135 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:51.333 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:14.156 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:56.354 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:14.168 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:56.366 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:14.177 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:56.375 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:14.180 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:56.324 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:14.184 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:56.382 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:14.187 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:56.331 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:14.243 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:56.441 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:14.255 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:56.454 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:14.257 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:15:56.401 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:24.175 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:06.373 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:24.177 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:06.375 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:24.199 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:06.397 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:24.226 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:06.370 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:24.227 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:06.371 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:24.248 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:06.446 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:24.248 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:06.392 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:24.249 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:06.447 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:24.272 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:06.470 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:30.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (21449 ms) 2023-11-25T05:19:30.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2023-11-25T05:19:30.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (24891 ms) 2023-11-25T05:19:30.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2023-11-25T05:19:30.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (4401 ms) 2023-11-25T05:19:30.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2023-11-25T05:19:30.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:19:30.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3434 ms) 2023-11-25T05:19:30.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2023-11-25T05:19:30.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4638 ms) 2023-11-25T05:19:30.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2023-11-25T05:19:30.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:19:30.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4504 ms) 2023-11-25T05:19:30.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2023-11-25T05:19:30.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:19:30.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5494 ms) 2023-11-25T05:19:30.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2023-11-25T05:19:30.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:19:30.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40842 ms) 2023-11-25T05:19:30.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2023-11-25T05:19:30.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:19:30.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17635 ms) 2023-11-25T05:19:30.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2023-11-25T05:19:30.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:19:30.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5373 ms) 2023-11-25T05:19:30.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2023-11-25T05:19:30.213 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:19:30.213 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4287 ms) 2023-11-25T05:19:30.213 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2023-11-25T05:19:30.213 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:19:30.213 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (26837 ms) 2023-11-25T05:19:30.213 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2023-11-25T05:19:30.213 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:19:30.213 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (16965 ms) 2023-11-25T05:19:30.213 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2023-11-25T05:19:30.213 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:19:30.213 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6835 ms) 2023-11-25T05:19:30.213 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2023-11-25T05:19:30.213 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:19:30.213 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5387 ms) 2023-11-25T05:19:30.214 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2023-11-25T05:19:30.214 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:19:30.214 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: waiting for scrubs... 2023-11-25T05:19:30.214 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: done waiting 2023-11-25T05:19:30.214 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26228 ms) 2023-11-25T05:19:30.214 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2023-11-25T05:19:30.214 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:19:30.214 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (11282 ms) 2023-11-25T05:19:30.214 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2023-11-25T05:19:30.214 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:19:30.214 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (10843 ms) 2023-11-25T05:19:30.214 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2023-11-25T05:19:30.214 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:19:30.215 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9986 ms) 2023-11-25T05:19:30.215 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2023-11-25T05:19:39.104 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:21.302 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:39.107 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:21.306 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:39.134 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:21.332 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:39.147 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:21.291 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:39.150 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:21.294 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:39.155 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:21.353 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:39.157 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:21.356 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:39.175 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:21.319 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:39.184 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:21.383 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:44.234 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:26.432 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:44.237 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:26.435 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:44.254 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:26.452 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:44.279 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:26.423 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:44.281 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:26.425 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:44.293 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:26.491 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:44.295 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:26.493 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:44.298 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:26.442 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:44.312 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:26.510 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:54.373 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:36.571 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:54.376 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:36.574 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:54.410 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:36.608 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:54.442 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:36.637 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:54.442 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:36.583 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:54.442 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:36.586 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:54.449 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:36.647 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:54.472 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:36.616 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:19:54.483 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:36.682 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:04.217 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:46.413 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:04.218 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:46.415 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:04.231 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:46.429 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:04.270 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:46.468 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:04.270 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:46.415 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:04.271 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:46.469 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:04.272 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:46.416 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:04.286 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:46.430 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:04.292 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:46.489 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:14.402 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:56.601 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:14.405 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:56.603 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:14.428 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:56.626 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:14.447 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:56.592 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:14.451 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:56.594 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:14.461 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:56.659 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:14.466 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:16:56.610 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:14.467 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:56.665 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:14.487 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:56.685 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:29.321 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:11.519 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:29.324 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:11.522 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:29.351 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:11.549 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:29.373 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:11.517 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:29.375 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:11.520 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:29.376 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:11.575 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:29.378 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:11.576 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:29.396 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:11.540 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:29.399 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:11.597 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:30.013 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:12.157 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:30.013 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:12.158 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:30.022 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:12.165 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:30.086 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:12.284 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:30.087 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:12.285 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:30.101 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:12.299 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:30.106 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:12.304 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:30.108 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:12.305 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:30.116 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:12.314 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:32.907 INFO:tasks.ceph.osd.6.smithi090.stderr:2023-11-25T05:20:32.892+0000 23a7f640 -1 snap_mapper.add_oid found existing snaps mapped on 51:c73f86bd:test-rados-api-smithi007-92908-97::foo:32, removing 2023-11-25T05:20:32.965 INFO:tasks.ceph.osd.5.smithi090.stderr:2023-11-25T05:20:32.954+0000 27e87640 -1 snap_mapper.add_oid found existing snaps mapped on 51:c73f86bd:test-rados-api-smithi007-92908-97::foo:32, removing 2023-11-25T05:20:44.507 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:26.705 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:44.512 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:26.710 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:44.538 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:26.737 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:44.559 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:26.703 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:44.560 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:26.704 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:44.563 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:26.762 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:44.568 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:26.766 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:44.586 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:26.730 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:44.596 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:26.794 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:49.425 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:31.623 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:49.426 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:31.625 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:49.440 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:31.638 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:49.473 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:31.617 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:49.474 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:31.618 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:49.478 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:31.676 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:49.483 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:31.681 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:49.489 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:31.633 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:49.496 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:31.694 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:59.464 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:41.662 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:59.465 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:41.664 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:59.488 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:41.686 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:59.504 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:41.648 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:59.509 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:41.653 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:59.532 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:41.676 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:59.532 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:41.730 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:59.533 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:41.732 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:20:59.559 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:41.757 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:09.506 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:51.705 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:09.508 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:51.706 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:09.531 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:51.729 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:09.555 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:51.699 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:09.556 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:51.700 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:09.572 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:51.771 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:09.574 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:51.773 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:09.579 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:51.723 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:09.600 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:51.798 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:14.659 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:56.858 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:14.723 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:56.922 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:14.748 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:56.947 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:14.768 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:56.912 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:14.769 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:56.914 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:14.784 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:17:56.928 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:14.794 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:56.992 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:14.795 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:56.993 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:14.817 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:57.015 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:17.888 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:00.087 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:17.892 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:00.090 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:17.913 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:00.111 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:17.940 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:00.084 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:17.943 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:00.087 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:17.949 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:00.147 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:17.952 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:00.150 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:17.958 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:00.103 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:17.970 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:00.168 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:29.754 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:11.952 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:29.756 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:11.955 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:29.784 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:11.982 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:29.802 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:12.001 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:29.807 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:12.005 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:29.808 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:11.952 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:29.812 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:11.956 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:29.835 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:12.033 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:29.840 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:11.984 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:30.571 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:12.716 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:30.572 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:12.717 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:30.581 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:12.726 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:30.658 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:12.856 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:30.659 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:12.858 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:30.669 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:12.868 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:30.684 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:12.882 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:30.686 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:12.884 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:30.697 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:12.895 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:44.874 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:27.072 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:44.878 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:27.076 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:44.910 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:27.108 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:44.934 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:27.076 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:44.934 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:27.078 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:44.941 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:27.140 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:44.946 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:27.144 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:44.976 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:27.120 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:44.989 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:27.188 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:49.791 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:31.989 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:49.796 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:31.994 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:49.812 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:32.010 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:49.842 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:31.986 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:49.844 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:31.988 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:49.851 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:32.049 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:49.853 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:32.051 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:49.858 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:32.002 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:49.874 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:32.073 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:59.817 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:42.016 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:59.820 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:42.019 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:59.848 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:42.046 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:59.866 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:42.010 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:59.868 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:42.012 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:59.891 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:42.035 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:59.896 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:42.094 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:59.897 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:42.096 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:21:59.926 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:42.124 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:09.875 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:52.073 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:09.876 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:52.075 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:09.914 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:52.111 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:09.917 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:52.061 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:09.924 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:52.064 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:09.942 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:52.086 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:09.949 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:52.147 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:09.968 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:52.166 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:09.992 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:52.190 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:14.895 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:57.092 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:14.895 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:57.093 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:14.906 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:57.104 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:14.940 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:57.083 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:14.941 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:57.085 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:14.946 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:57.144 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:14.948 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:57.146 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:14.952 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:18:57.096 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:14.959 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:57.157 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:29.986 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:12.185 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:29.992 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:12.189 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:30.008 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:12.207 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:30.036 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:12.180 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:30.038 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:12.181 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:30.056 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:12.198 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:30.059 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:12.253 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:30.060 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:12.255 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:30.076 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:12.272 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:35.065 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:17.263 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:35.068 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:17.267 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:35.071 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:17.269 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:35.071 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:17.215 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:35.073 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:17.271 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:35.120 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:17.263 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:35.135 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:17.279 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:35.136 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:17.335 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:35.141 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:17.339 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:50.335 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:32.533 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:50.338 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:32.536 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:50.385 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:32.583 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:50.397 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:32.541 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:50.401 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:32.545 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:50.402 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:32.600 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:50.413 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:32.611 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:50.444 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:32.587 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:22:50.460 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:32.658 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:05.150 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:47.348 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:05.151 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:47.349 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:05.184 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:47.382 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:05.196 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:47.341 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:05.198 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:47.342 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:05.202 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:47.401 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:05.205 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:47.403 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:05.225 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:47.369 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:05.234 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:47.432 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:15.330 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:57.528 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:15.332 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:57.531 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:15.355 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:57.554 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:15.388 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:57.586 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:15.389 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:57.587 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:15.391 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:57.535 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:15.398 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:57.542 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:15.417 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:57.615 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:15.423 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:57.567 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:16.464 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:58.662 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:16.467 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:58.665 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:16.486 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:58.685 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:16.521 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:58.665 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:16.524 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:58.668 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:16.529 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:58.727 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:16.530 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:58.729 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:16.539 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:19:58.683 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:16.545 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:58.744 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:19.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:23:19.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9950 ms) 2023-11-25T05:23:19.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2023-11-25T05:23:19.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:23:19.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (7623 ms) 2023-11-25T05:23:19.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2023-11-25T05:23:19.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:23:19.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6658 ms) 2023-11-25T05:23:19.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2023-11-25T05:23:19.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:23:19.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26292 ms) 2023-11-25T05:23:19.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2023-11-25T05:23:19.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:23:19.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14923 ms) 2023-11-25T05:23:19.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2023-11-25T05:23:19.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12122 ms) 2023-11-25T05:23:19.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2023-11-25T05:23:19.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2023-11-25T05:23:19.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: 00000010 69 73 |is| 2023-11-25T05:23:19.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: 00000012 2023-11-25T05:23:19.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (11822 ms) 2023-11-25T05:23:19.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2023-11-25T05:23:19.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = reef 2023-11-25T05:23:19.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6378 ms) 2023-11-25T05:23:19.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (885050 ms total) 2023-11-25T05:23:19.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: 2023-11-25T05:23:19.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2023-11-25T05:23:19.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2023-11-25T05:23:19.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1379 ms) 2023-11-25T05:23:19.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2023-11-25T05:23:19.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10678 ms) 2023-11-25T05:23:19.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2023-11-25T05:23:19.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18948 ms) 2023-11-25T05:23:19.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2023-11-25T05:23:19.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (20 ms) 2023-11-25T05:23:19.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (31025 ms total) 2023-11-25T05:23:19.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: 2023-11-25T05:23:19.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2023-11-25T05:23:19.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2023-11-25T05:23:19.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7881 ms) 2023-11-25T05:23:19.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2023-11-25T05:23:19.657 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (9918 ms) 2023-11-25T05:23:19.657 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2023-11-25T05:23:19.657 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: waiting for scrub... 2023-11-25T05:23:19.657 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: done waiting 2023-11-25T05:23:19.657 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (26655 ms) 2023-11-25T05:23:19.657 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2023-11-25T05:23:19.657 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (11875 ms) 2023-11-25T05:23:19.657 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2023-11-25T05:23:19.657 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8702 ms) 2023-11-25T05:23:19.657 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2023-11-25T05:23:19.657 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (9017 ms) 2023-11-25T05:23:19.658 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2023-11-25T05:23:19.658 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (11667 ms) 2023-11-25T05:23:19.658 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2023-11-25T05:23:20.384 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:02.582 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:20.389 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:02.586 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:20.402 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:02.599 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:20.442 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:02.638 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:20.447 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:02.645 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:20.460 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:02.602 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:20.461 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:02.657 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:20.461 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:02.603 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:20.481 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:02.622 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:35.357 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:17.556 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:35.410 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:17.554 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:35.412 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:17.555 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:35.419 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:17.617 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:35.421 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:17.619 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:35.425 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:17.624 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:35.435 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:17.580 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:35.453 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:17.651 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:35.463 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:17.661 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:40.253 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:22.452 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:40.311 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:22.509 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:40.351 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:22.549 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:40.356 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:22.501 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:40.359 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:22.504 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:40.372 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:22.571 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:40.374 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:22.518 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:40.377 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:22.575 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:40.397 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:22.595 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:55.578 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:37.776 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:55.638 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:37.782 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:55.639 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:37.784 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:55.641 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:37.840 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:55.678 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:37.876 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:55.686 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:37.830 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:55.710 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:37.908 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:55.717 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:37.915 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:23:55.762 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:37.961 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:10.797 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:52.995 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:10.803 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:53.001 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:10.845 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:53.043 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:10.862 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:53.006 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:10.867 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:53.066 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:10.869 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:53.067 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:10.869 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:53.014 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:10.921 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:53.065 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:10.928 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:53.126 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:15.740 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:57.933 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:15.757 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:57.956 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:15.771 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:57.969 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:15.776 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:57.920 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:15.778 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:57.922 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:15.786 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:57.985 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:15.788 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:57.986 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:15.795 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:20:57.935 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:15.801 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:57.999 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:25.680 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:07.878 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:25.681 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:07.880 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:25.724 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:07.922 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:25.728 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:07.926 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:25.732 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:07.930 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:25.733 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:07.878 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:25.735 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:07.879 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:25.756 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:07.900 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:25.760 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:07.958 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:35.849 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:18.047 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:35.852 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:18.050 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:35.880 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:18.078 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:35.895 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:18.039 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:35.898 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:18.042 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:35.918 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:18.116 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:35.919 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:18.117 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:35.920 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:18.065 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:35.943 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:18.141 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:40.738 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:22.937 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:40.742 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:22.941 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:40.758 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:22.956 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:40.792 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:22.990 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:40.793 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:22.934 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:40.793 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:22.937 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:40.804 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:22.948 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:40.811 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:23.009 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:40.826 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:23.024 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:55.948 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:38.145 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:55.954 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:38.152 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:55.979 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:38.176 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:55.998 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:38.142 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:56.001 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:38.145 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:56.003 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:38.201 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:56.007 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:38.204 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:56.022 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:38.166 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:24:56.028 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:38.227 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:10.823 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:53.021 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:10.827 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:53.025 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:10.858 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:53.056 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:10.871 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:53.015 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:10.875 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:53.019 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:10.879 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:53.077 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:10.882 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:53.080 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:10.903 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:53.047 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:10.911 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:53.110 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:15.843 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:58.041 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:15.847 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:58.045 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:15.861 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:58.059 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:15.891 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:58.035 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:15.892 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:58.037 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:15.896 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:58.093 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:15.897 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:58.096 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:15.905 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:21:58.049 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:15.910 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:58.109 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:19.136 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:01.334 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:19.144 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:01.342 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:19.158 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:01.356 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:19.196 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:01.340 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:19.196 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:01.341 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:19.203 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:01.402 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:19.204 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:01.403 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:19.209 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:01.353 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:19.218 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:01.416 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:30.907 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:13.105 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:30.910 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:13.108 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:30.929 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:13.128 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:30.958 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:13.101 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:30.958 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:13.103 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:30.966 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:13.163 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:30.969 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:13.167 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:30.977 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:13.122 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:30.993 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:13.190 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:35.984 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:18.182 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:36.038 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:18.237 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:36.057 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:18.255 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:36.079 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:18.223 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:36.084 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:18.228 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:36.090 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:18.288 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:36.091 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:18.290 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:36.098 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:18.242 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:36.106 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:18.304 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:45.998 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:28.193 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:45.999 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:28.195 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:46.015 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:28.213 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:46.040 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:28.184 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:46.042 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:28.186 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:46.045 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:28.243 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:46.047 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:28.245 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:46.060 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:28.205 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:46.068 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:28.264 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:55.993 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:38.192 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:55.994 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:38.193 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:56.014 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:38.212 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:56.041 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:38.185 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:56.041 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:38.186 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:56.043 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:38.241 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:56.046 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:38.244 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:56.059 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:38.203 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:25:56.069 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:38.267 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:01.192 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:43.336 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:01.200 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:43.398 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:01.202 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:43.400 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:01.214 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:43.358 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:01.224 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:43.422 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:01.225 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:43.423 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:01.242 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:43.386 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:01.277 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:43.475 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:01.333 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:43.531 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:15.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (9136 ms) 2023-11-25T05:26:15.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2023-11-25T05:26:15.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14453 ms) 2023-11-25T05:26:15.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2023-11-25T05:26:15.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (9978 ms) 2023-11-25T05:26:15.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2023-11-25T05:26:15.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (15528 ms) 2023-11-25T05:26:15.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2023-11-25T05:26:15.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (9383 ms) 2023-11-25T05:26:15.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2023-11-25T05:26:15.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (9137 ms) 2023-11-25T05:26:15.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2023-11-25T05:26:15.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: hmm, no HitSet yet 2023-11-25T05:26:15.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: ok, hit_set contains 251:602f83fe:::foo:head 2023-11-25T05:26:15.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9856 ms) 2023-11-25T05:26:15.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2023-11-25T05:26:15.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889876,0 2023-11-25T05:26:15.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: first is 1700889876 2023-11-25T05:26:15.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889876,0 2023-11-25T05:26:15.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889876,0 2023-11-25T05:26:15.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889876,1700889878,1700889879,0 2023-11-25T05:26:15.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889876,1700889878,1700889879,0 2023-11-25T05:26:15.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889876,1700889878,1700889879,0 2023-11-25T05:26:15.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889876,1700889878,1700889879,1700889881,1700889882,0 2023-11-25T05:26:15.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889876,1700889878,1700889879,1700889881,1700889882,0 2023-11-25T05:26:15.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889876,1700889878,1700889879,1700889881,1700889882,0 2023-11-25T05:26:15.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889876,1700889878,1700889879,1700889881,1700889882,1700889884,1700889885,0 2023-11-25T05:26:15.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889876,1700889878,1700889879,1700889881,1700889882,1700889884,1700889885,0 2023-11-25T05:26:15.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889876,1700889878,1700889879,1700889881,1700889882,1700889884,1700889885,0 2023-11-25T05:26:15.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1700889879,1700889881,1700889882,1700889884,1700889885,1700889888,1700889889,0 2023-11-25T05:26:15.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: first now 1700889879, trimmed 2023-11-25T05:26:15.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22196 ms) 2023-11-25T05:26:15.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2023-11-25T05:26:15.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: foo0 2023-11-25T05:26:15.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-11-25T05:26:15.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (16734 ms) 2023-11-25T05:26:15.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2023-11-25T05:26:15.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (20362 ms) 2023-11-25T05:26:15.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2023-11-25T05:26:15.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (24733 ms) 2023-11-25T05:26:15.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2023-11-25T05:26:15.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (6770 ms) 2023-11-25T05:26:15.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2023-11-25T05:26:15.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4675 ms) 2023-11-25T05:26:15.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2023-11-25T05:26:15.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4542 ms) 2023-11-25T05:26:15.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2023-11-25T05:26:15.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4550 ms) 2023-11-25T05:26:15.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (267748 ms total) 2023-11-25T05:26:15.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: 2023-11-25T05:26:15.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] Global test environment tear-down 2023-11-25T05:26:15.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1240232 ms total) 2023-11-25T05:26:15.887 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ PASSED ] 77 tests. 2023-11-25T05:26:15.890 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.890 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92668 2023-11-25T05:26:15.890 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 92668 2023-11-25T05:26:15.890 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.890 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92854 2023-11-25T05:26:15.890 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 92854 2023-11-25T05:26:15.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93372 2023-11-25T05:26:15.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 93372 2023-11-25T05:26:15.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93126 2023-11-25T05:26:15.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 93126 2023-11-25T05:26:15.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93006 2023-11-25T05:26:15.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 93006 2023-11-25T05:26:15.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92754 2023-11-25T05:26:15.891 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 92754 2023-11-25T05:26:15.892 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.892 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93082 2023-11-25T05:26:15.892 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 93082 2023-11-25T05:26:15.892 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.892 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92768 2023-11-25T05:26:15.892 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 92768 2023-11-25T05:26:15.892 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.892 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93294 2023-11-25T05:26:15.892 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 93294 2023-11-25T05:26:15.892 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.892 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92640 2023-11-25T05:26:15.892 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 92640 2023-11-25T05:26:15.893 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.893 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92708 2023-11-25T05:26:15.893 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 92708 2023-11-25T05:26:15.893 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.893 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93333 2023-11-25T05:26:15.893 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 93333 2023-11-25T05:26:15.893 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.893 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92631 2023-11-25T05:26:15.893 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 92631 2023-11-25T05:26:15.893 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.893 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92653 2023-11-25T05:26:15.894 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 92653 2023-11-25T05:26:15.894 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.894 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92902 2023-11-25T05:26:15.894 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 92902 2023-11-25T05:26:15.894 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.894 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93060 2023-11-25T05:26:15.894 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 93060 2023-11-25T05:26:15.894 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.894 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92976 2023-11-25T05:26:15.894 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 92976 2023-11-25T05:26:15.894 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.894 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93345 2023-11-25T05:26:15.895 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 93345 2023-11-25T05:26:15.895 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.895 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93238 2023-11-25T05:26:15.895 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 93238 2023-11-25T05:26:15.895 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.895 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92811 2023-11-25T05:26:15.895 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 92811 2023-11-25T05:26:15.895 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.895 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93157 2023-11-25T05:26:15.895 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 93157 2023-11-25T05:26:15.895 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.896 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92624 2023-11-25T05:26:15.896 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 92624 2023-11-25T05:26:15.896 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.896 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92685 2023-11-25T05:26:15.896 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 92685 2023-11-25T05:26:15.896 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.896 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93094 2023-11-25T05:26:15.896 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 93094 2023-11-25T05:26:15.896 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.896 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=92736 2023-11-25T05:26:15.896 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 92736 2023-11-25T05:26:15.896 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.896 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93192 2023-11-25T05:26:15.897 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 93192 2023-11-25T05:26:15.897 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2023-11-25T05:26:15.897 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=93222 2023-11-25T05:26:15.897 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 93222 2023-11-25T05:26:15.897 INFO:tasks.workunit.client.0.smithi007.stderr:+ exit 0 2023-11-25T05:26:15.897 INFO:tasks.workunit.client.0.smithi007.stderr:+ cleanup 2023-11-25T05:26:15.897 INFO:tasks.workunit.client.0.smithi007.stderr:+ pkill -P 92619 2023-11-25T05:26:15.901 INFO:tasks.workunit.client.0.smithi007.stderr:+ true 2023-11-25T05:26:15.902 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-25T05:26:15.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-25T05:26:15.976 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2023-11-25T05:26:15.976 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-11-25T05:26:16.220 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:58.418 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:16.223 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:58.421 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:16.251 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:58.450 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:16.269 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:58.413 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:16.270 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:58.414 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:16.297 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:22:58.441 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:16.298 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:58.496 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:16.300 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:58.499 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:16.346 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:58.544 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:16.352 DEBUG:teuthology.parallel:result is None 2023-11-25T05:26:16.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-11-25T05:26:16.384 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-11-25T05:26:16.384 DEBUG:teuthology.orchestra.run.smithi007:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-11-25T05:26:16.449 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-11-25T05:26:16.449 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-25T05:26:16.458 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-25T05:26:16.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-25T05:26:16.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.595+0000 7fd08d941640 10 monclient: get_monmap_and_config 2023-11-25T05:26:16.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.595+0000 7fd08d941640 10 monclient: build_initial_monmap 2023-11-25T05:26:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.595+0000 7fd08d941640 10 monclient: monmap: 2023-11-25T05:26:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:16.596289+0000 2023-11-25T05:26:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:16.596289+0000 2023-11-25T05:26:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:16.598 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.595+0000 7fd08d941640 1 Processor -- start 2023-11-25T05:26:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 1 -- start start 2023-11-25T05:26:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 10 monclient: init 2023-11-25T05:26:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2023-11-25T05:26:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd0880af360 0x7fd0881430e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 10 monclient(hunting): picked mon.noname-b con 0x7fd0880af360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd088143620 addr v1:172.21.15.90:6789/0 2023-11-25T05:26:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd08806e7a0 0x7fd08814a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 10 monclient(hunting): picked mon.noname-c con 0x7fd08806e7a0 addr v2:172.21.15.90:3300/0 2023-11-25T05:26:16.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd088071cb0 con 0x7fd088143620 2023-11-25T05:26:16.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:16.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0880a8790 con 0x7fd08806e7a0 2023-11-25T05:26:16.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:16.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0880af740 con 0x7fd0880af360 2023-11-25T05:26:16.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:16.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.596+0000 7fd08d941640 10 monclient(hunting): authenticate will time out at 2702.839498s 2023-11-25T05:26:16.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.608+0000 7fd086ffd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd0880af360 0x7fd0881430e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:16.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.610+0000 7fd0877fe640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd08806e7a0 0x7fd08814a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:16.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.614+0000 7fd0867fc640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd088143620 0x7fd08806e090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:40482/0 (socket says 172.21.15.7:40482) 2023-11-25T05:26:16.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.614+0000 7fd0867fc640 1 -- 172.21.15.7:0/3010602230 learned_addr learned my addr 172.21.15.7:0/3010602230 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:26:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.622+0000 7fd0877fe640 10 monclient(hunting): get_auth_request con 0x7fd08806e7a0 auth_method 0 2023-11-25T05:26:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.622+0000 7fd0877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.622+0000 7fd0877fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.622+0000 7fd0877fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.624+0000 7fd086ffd640 10 monclient(hunting): get_auth_request con 0x7fd0880af360 auth_method 0 2023-11-25T05:26:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.624+0000 7fd086ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.624+0000 7fd086ffd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.624+0000 7fd086ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:16.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.629+0000 7fd086ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:16.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.629+0000 7fd086ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:16.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.629+0000 7fd086ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.635+0000 7fd086ffd640 10 monclient(hunting): handle_auth_done global_id 17357 payload 274 2023-11-25T05:26:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.635+0000 7fd086ffd640 1 -- 172.21.15.7:0/3010602230 >> v2:172.21.15.90:3300/0 conn(0x7fd08806e7a0 msgr2=0x7fd08814a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.635+0000 7fd086ffd640 1 --2- 172.21.15.7:0/3010602230 >> v2:172.21.15.90:3300/0 conn(0x7fd08806e7a0 0x7fd08814a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.635+0000 7fd086ffd640 1 -- 172.21.15.7:0/3010602230 >> v1:172.21.15.90:6789/0 conn(0x7fd088143620 legacy=0x7fd08806e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.635+0000 7fd086ffd640 10 monclient: _finish_hunting 0 2023-11-25T05:26:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.635+0000 7fd086ffd640 1 monclient: found mon.noname-b 2023-11-25T05:26:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.635+0000 7fd086ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.635+0000 7fd086ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:26:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.635+0000 7fd086ffd640 1 -- 172.21.15.7:0/3010602230 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd088072140 con 0x7fd0880af360 2023-11-25T05:26:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.638+0000 7fd086ffd640 1 --2- 172.21.15.7:0/3010602230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd0880af360 0x7fd0881430e0 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7fd0780097a0 tx=0x7fd07802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0fa7326315573e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:16.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd085ffb640 1 -- 172.21.15.7:0/3010602230 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd078031830 con 0x7fd0880af360 2023-11-25T05:26:16.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd085ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd085ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd085ffb640 10 monclient: dump: 2023-11-25T05:26:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:16.669 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:16.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd085ffb640 10 monclient: _finish_auth 0 2023-11-25T05:26:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd085ffb640 10 monclient: _check_auth_tickets 2023-11-25T05:26:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd085ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd085ffb640 1 -- 172.21.15.7:0/3010602230 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd078031990 con 0x7fd0880af360 2023-11-25T05:26:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd085ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd085ffb640 1 -- 172.21.15.7:0/3010602230 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd078031d20 con 0x7fd0880af360 2023-11-25T05:26:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd085ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd085ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:26:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd085ffb640 10 monclient: dump: 2023-11-25T05:26:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:16.670 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:16.671 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:16.671 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:16.671 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:16.671 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:16.671 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:16.671 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:16.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd08d941640 5 monclient: authenticate success, global_id 17357 2023-11-25T05:26:16.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd08d941640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:16.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.667+0000 7fd08d941640 4 set_mon_vals no callback set 2023-11-25T05:26:16.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.668+0000 7fd08d941640 10 monclient: shutdown 2023-11-25T05:26:16.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.668+0000 7fd08d941640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:16.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.668+0000 7fd08d941640 1 -- 172.21.15.7:0/3010602230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd0880af360 msgr2=0x7fd0881430e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:16.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.668+0000 7fd08d941640 1 --2- 172.21.15.7:0/3010602230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd0880af360 0x7fd0881430e0 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7fd0780097a0 tx=0x7fd07802b8d0 comp rx=0 tx=0).stop 2023-11-25T05:26:16.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.668+0000 7fd08d941640 1 -- 172.21.15.7:0/3010602230 shutdown_connections 2023-11-25T05:26:16.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.668+0000 7fd08d941640 1 --2- 172.21.15.7:0/3010602230 >> v2:172.21.15.90:3300/0 conn(0x7fd08806e7a0 0x7fd08814a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:16.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.668+0000 7fd08d941640 1 --2- 172.21.15.7:0/3010602230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd0880af360 0x7fd0881430e0 unknown :-1 s=CLOSED pgs=3060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:16.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.668+0000 7fd08d941640 1 -- 172.21.15.7:0/3010602230 >> 172.21.15.7:0/3010602230 conn(0x7fd0880b01e0 msgr2=0x7fd0880b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:16.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.668+0000 7fd08d941640 1 -- 172.21.15.7:0/3010602230 shutdown_connections 2023-11-25T05:26:16.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.668+0000 7fd08d941640 1 -- 172.21.15.7:0/3010602230 wait complete. 2023-11-25T05:26:16.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.669+0000 7fd08d941640 10 monclient: build_initial_monmap 2023-11-25T05:26:16.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.669+0000 7fd08d941640 10 monclient: monmap: 2023-11-25T05:26:16.675 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:16.675 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:16.675 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:16.675 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:16.675 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:16.675 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:16.675 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:16.675 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:16.676 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:16.676 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:16.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.669+0000 7fd08d941640 1 librados: starting msgr at 2023-11-25T05:26:16.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.669+0000 7fd08d941640 1 librados: starting objecter 2023-11-25T05:26:16.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.669+0000 7fd08d941640 1 Processor -- start 2023-11-25T05:26:16.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.669+0000 7fd08d941640 1 -- start start 2023-11-25T05:26:16.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.669+0000 7fd08d941640 1 librados: setting wanted keys 2023-11-25T05:26:16.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.669+0000 7fd08d941640 1 librados: calling monclient init 2023-11-25T05:26:16.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.669+0000 7fd08d941640 10 monclient: init 2023-11-25T05:26:16.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:16.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:26:16.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd08806e7a0 0x7fd0880ac980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:16.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 10 monclient(hunting): picked mon.noname-c con 0x7fd08806e7a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:16.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0880af360 0x7fd0881402b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:16.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 10 monclient(hunting): picked mon.noname-b con 0x7fd0880af360 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:16.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd088143620 0x7fd088136950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:16.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 10 monclient(hunting): picked mon.noname-a con 0x7fd088143620 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:16.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:16.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0881407f0 con 0x7fd088143620 2023-11-25T05:26:16.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:16.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd088140a10 con 0x7fd0880af360 2023-11-25T05:26:16.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:16.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08813fe10 con 0x7fd08806e7a0 2023-11-25T05:26:16.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:16.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.670+0000 7fd08d941640 10 monclient(hunting): authenticate will time out at 2702.913345s 2023-11-25T05:26:16.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.672+0000 7fd0877fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd088143620 0x7fd088136950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:16.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.673+0000 7fd086ffd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd08806e7a0 0x7fd0880ac980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:16.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.674+0000 7fd0877fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd088143620 0x7fd088136950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43598/0 (socket says 172.21.15.7:43598) 2023-11-25T05:26:16.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.674+0000 7fd0877fe640 1 -- 172.21.15.7:0/1430558470 learned_addr learned my addr 172.21.15.7:0/1430558470 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:16.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.674+0000 7fd0877fe640 10 monclient(hunting): get_auth_request con 0x7fd088143620 auth_method 0 2023-11-25T05:26:16.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.674+0000 7fd0877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:16.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.674+0000 7fd0877fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:16.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.674+0000 7fd0877fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:16.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.675+0000 7fd0877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:16.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.675+0000 7fd0877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:16.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.675+0000 7fd0877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:16.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.676+0000 7fd086ffd640 10 monclient(hunting): get_auth_request con 0x7fd08806e7a0 auth_method 0 2023-11-25T05:26:16.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.676+0000 7fd086ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:16.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.676+0000 7fd086ffd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:16.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.676+0000 7fd086ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:16.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.677+0000 7fd0867fc640 1 --2- 172.21.15.7:0/1430558470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0880af360 0x7fd0881402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:16.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.678+0000 7fd0867fc640 10 monclient(hunting): get_auth_request con 0x7fd0880af360 auth_method 0 2023-11-25T05:26:16.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.678+0000 7fd0867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:16.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.678+0000 7fd0867fc640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:16.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.678+0000 7fd0867fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:16.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.678+0000 7fd086ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:16.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.678+0000 7fd086ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:16.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.678+0000 7fd086ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:16.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.679+0000 7fd0877fe640 10 monclient(hunting): handle_auth_done global_id 15888 payload 824 2023-11-25T05:26:16.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.679+0000 7fd0877fe640 1 -- 172.21.15.7:0/1430558470 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd08806e7a0 msgr2=0x7fd0880ac980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:16.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.679+0000 7fd0877fe640 1 --2- 172.21.15.7:0/1430558470 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd08806e7a0 0x7fd0880ac980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:16.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.679+0000 7fd0877fe640 1 -- 172.21.15.7:0/1430558470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0880af360 msgr2=0x7fd0881402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:16.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.679+0000 7fd0877fe640 1 --2- 172.21.15.7:0/1430558470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0880af360 0x7fd0881402b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:16.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.679+0000 7fd0877fe640 10 monclient: _finish_hunting 0 2023-11-25T05:26:16.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.679+0000 7fd0877fe640 1 monclient: found mon.noname-a 2023-11-25T05:26:16.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.679+0000 7fd0877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:16.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.679+0000 7fd0877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:26:16.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.679+0000 7fd0877fe640 1 -- 172.21.15.7:0/1430558470 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd078009400 con 0x7fd088143620 2023-11-25T05:26:16.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.686+0000 7fd0877fe640 1 --2- 172.21.15.7:0/1430558470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd088143620 0x7fd088136950 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7fd080007e40 tx=0x7fd08000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:16.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.686+0000 7fd06ffff640 10 client.?.objecter ms_handle_connect 0x7fd088143620 2023-11-25T05:26:16.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.686+0000 7fd06ffff640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:16.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.697+0000 7fd06ffff640 1 -- 172.21.15.7:0/1430558470 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd080014620 con 0x7fd088143620 2023-11-25T05:26:16.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd06ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:16.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd06ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:26:16.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd06ffff640 10 monclient: dump: 2023-11-25T05:26:16.700 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:16.700 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:16.700 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:16.700 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:16.700 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:16.700 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:16.700 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:16.700 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:16.700 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:16.700 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:16.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd06ffff640 10 monclient: _finish_auth 0 2023-11-25T05:26:16.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd06ffff640 10 monclient: _check_auth_tickets 2023-11-25T05:26:16.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd06ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:16.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd06ffff640 1 -- 172.21.15.7:0/1430558470 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08000ee90 con 0x7fd088143620 2023-11-25T05:26:16.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd06ffff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:16.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd06ffff640 1 -- 172.21.15.7:0/1430558470 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0800149e0 con 0x7fd088143620 2023-11-25T05:26:16.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd08d941640 5 monclient: authenticate success, global_id 15888 2023-11-25T05:26:16.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd08d941640 10 monclient: _renew_subs 2023-11-25T05:26:16.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd08d941640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:26:16.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd08d941640 1 -- 172.21.15.7:0/1430558470 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd088140090 con 0x7fd088143620 2023-11-25T05:26:16.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd0857fa640 4 set_mon_vals no callback set 2023-11-25T05:26:16.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd06ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:16.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd06ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:26:16.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd06ffff640 10 monclient: dump: 2023-11-25T05:26:16.701 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:16.702 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:16.702 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:16.702 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:16.702 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:16.702 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:16.702 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:16.702 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:16.702 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:16.703 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:16.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd08d941640 10 client.15888.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:16.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd08d941640 10 monclient: _renew_subs 2023-11-25T05:26:16.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd08d941640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:26:16.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd08d941640 1 -- 172.21.15.7:0/1430558470 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd088137070 con 0x7fd088143620 2023-11-25T05:26:16.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.698+0000 7fd08d941640 1 librados: init done 2023-11-25T05:26:16.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.699+0000 7fd08d941640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:16.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.699+0000 7fd08d941640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:16.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.699+0000 7fd08d941640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:26:16.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.699+0000 7fd08d941640 1 -- 172.21.15.7:0/1430558470 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd088075b60 con 0x7fd088143620 2023-11-25T05:26:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.970+0000 7fd06ffff640 1 -- 172.21.15.7:0/1430558470 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd080014b40 con 0x7fd088143620 2023-11-25T05:26:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.971+0000 7fd06ffff640 1 --2- 172.21.15.7:0/1430558470 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fd07406c800 0x7fd07406ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.971+0000 7fd086ffd640 1 --2- 172.21.15.7:0/1430558470 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fd07406c800 0x7fd07406ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:16.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.971+0000 7fd086ffd640 10 monclient: get_auth_request con 0x7fd07406c800 auth_method 0 2023-11-25T05:26:16.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.971+0000 7fd086ffd640 1 --2- 172.21.15.7:0/1430558470 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fd07406c800 0x7fd07406ecc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd0780097a0 tx=0x7fd07803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:16.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.971+0000 7fd06ffff640 10 client.15888.objecter ms_handle_connect 0x7fd07406c800 2023-11-25T05:26:16.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.989+0000 7fd06ffff640 1 -- 172.21.15.7:0/1430558470 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(918..918 src has 1..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7fd08005a250 con 0x7fd088143620 2023-11-25T05:26:16.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.989+0000 7fd06ffff640 10 client.15888.objecter ms_dispatch 0x7fd088000b90 osd_map(918..918 src has 1..918) v4 2023-11-25T05:26:16.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.989+0000 7fd06ffff640 3 client.15888.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:16.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.989+0000 7fd06ffff640 3 client.15888.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:16.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.989+0000 7fd06ffff640 20 client.15888.objecter dump_active .. 0 homeless 2023-11-25T05:26:16.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.989+0000 7fd06ffff640 1 -- 172.21.15.7:0/1430558470 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd080055e90 con 0x7fd088143620 2023-11-25T05:26:16.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.989+0000 7fd06ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:16.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:16.989+0000 7fd06ffff640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:26:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.199+0000 7fd08d941640 1 -- 172.21.15.7:0/1430558470 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd088143a60 con 0x7fd07406c800 2023-11-25T05:26:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.205+0000 7fd06ffff640 1 -- 172.21.15.7:0/1430558470 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151923 (secure 0 0 0) 0x7fd07803a420 con 0x7fd07406c800 2023-11-25T05:26:17.208 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:26:17.209 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-25T05:26:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 10 librados: watch_flush enter 2023-11-25T05:26:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 10 librados: watch_flush exit 2023-11-25T05:26:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 20 client.15888.objecter shutdown clearing up homeless session... 2023-11-25T05:26:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 10 client.15888.objecter successfully canceled tick 2023-11-25T05:26:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 1 -- 172.21.15.7:0/1430558470 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fd07406c800 msgr2=0x7fd07406ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 1 --2- 172.21.15.7:0/1430558470 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fd07406c800 0x7fd07406ecc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd0780097a0 tx=0x7fd07803c040 comp rx=0 tx=0).stop 2023-11-25T05:26:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 10 monclient: shutdown 2023-11-25T05:26:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 1 -- 172.21.15.7:0/1430558470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd088143620 msgr2=0x7fd088136950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 1 --2- 172.21.15.7:0/1430558470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd088143620 0x7fd088136950 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7fd080007e40 tx=0x7fd08000e4b0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 1 -- 172.21.15.7:0/1430558470 shutdown_connections 2023-11-25T05:26:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 1 --2- 172.21.15.7:0/1430558470 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fd07406c800 0x7fd07406ecc0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 1 --2- 172.21.15.7:0/1430558470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd088143620 0x7fd088136950 unknown :-1 s=CLOSED pgs=3144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 1 --2- 172.21.15.7:0/1430558470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0880af360 0x7fd0881402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 1 --2- 172.21.15.7:0/1430558470 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd08806e7a0 0x7fd0880ac980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 1 -- 172.21.15.7:0/1430558470 >> 172.21.15.7:0/1430558470 conn(0x7fd0880b01e0 msgr2=0x7fd08813c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 1 -- 172.21.15.7:0/1430558470 shutdown_connections 2023-11-25T05:26:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.207+0000 7fd08d941640 1 -- 172.21.15.7:0/1430558470 wait complete. 2023-11-25T05:26:17.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.208+0000 7fd08d941640 1 librados: shutdown 2023-11-25T05:26:17.221 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1579,"stamp":"2023-11-25T05:26:16.098400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":596656,"kb_used_data":106952,"kb_used_omap":12,"kb_used_meta":489459,"kb_avail":754378064,"statfs":{"total":773094113280,"available":772483137536,"internally_reserved":0,"allocated":109518848,"data_stored":67197159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":501206579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":356,"apply_latency_ms":356,"commit_latency_ns":356000000,"apply_latency_ns":356000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.628323"},"pg_stats":[{"pgid":"9.15","version":"0'0","reported_seq":856,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.329954+0000","last_change":"2023-11-25T05:05:46.884752+0000","last_active":"2023-11-25T05:26:10.329954+0000","last_peered":"2023-11-25T05:26:10.329954+0000","last_clean":"2023-11-25T05:26:10.329954+0000","last_became_active":"2023-11-25T05:05:46.875394+0000","last_became_peered":"2023-11-25T05:05:46.875394+0000","last_unstale":"2023-11-25T05:26:10.329954+0000","last_undegraded":"2023-11-25T05:26:10.329954+0000","last_fullsized":"2023-11-25T05:26:10.329954+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:08:59.865717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":867,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.459091+0000","last_change":"2023-11-25T05:05:46.929874+0000","last_active":"2023-11-25T05:26:10.459091+0000","last_peered":"2023-11-25T05:26:10.459091+0000","last_clean":"2023-11-25T05:26:10.459091+0000","last_became_active":"2023-11-25T05:05:46.923663+0000","last_became_peered":"2023-11-25T05:05:46.923663+0000","last_unstale":"2023-11-25T05:26:10.459091+0000","last_undegraded":"2023-11-25T05:26:10.459091+0000","last_fullsized":"2023-11-25T05:26:10.459091+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:38:50.120101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.706495+0000","last_change":"2023-11-25T05:05:51.087897+0000","last_active":"2023-11-25T05:26:12.706495+0000","last_peered":"2023-11-25T05:26:12.706495+0000","last_clean":"2023-11-25T05:26:12.706495+0000","last_became_active":"2023-11-25T05:05:50.968719+0000","last_became_peered":"2023-11-25T05:05:50.968719+0000","last_unstale":"2023-11-25T05:26:12.706495+0000","last_undegraded":"2023-11-25T05:26:12.706495+0000","last_fullsized":"2023-11-25T05:26:12.706495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:28:52.119629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":846,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-11-25T05:26:09.285975+0000","last_change":"2023-11-25T05:06:05.094337+0000","last_active":"2023-11-25T05:26:09.285975+0000","last_peered":"2023-11-25T05:26:09.285975+0000","last_clean":"2023-11-25T05:26:09.285975+0000","last_became_active":"2023-11-25T05:06:05.084111+0000","last_became_peered":"2023-11-25T05:06:05.084111+0000","last_unstale":"2023-11-25T05:26:09.285975+0000","last_undegraded":"2023-11-25T05:26:09.285975+0000","last_fullsized":"2023-11-25T05:26:09.285975+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T13:39:56.459513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":856,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.255038+0000","last_change":"2023-11-25T05:05:47.892163+0000","last_active":"2023-11-25T05:26:10.255038+0000","last_peered":"2023-11-25T05:26:10.255038+0000","last_clean":"2023-11-25T05:26:10.255038+0000","last_became_active":"2023-11-25T05:05:47.855679+0000","last_became_peered":"2023-11-25T05:05:47.855679+0000","last_unstale":"2023-11-25T05:26:10.255038+0000","last_undegraded":"2023-11-25T05:26:10.255038+0000","last_fullsized":"2023-11-25T05:26:10.255038+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:54:11.446291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":874,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.189467+0000","last_change":"2023-11-25T05:06:09.646267+0000","last_active":"2023-11-25T05:26:14.189467+0000","last_peered":"2023-11-25T05:26:14.189467+0000","last_clean":"2023-11-25T05:26:14.189467+0000","last_became_active":"2023-11-25T05:06:09.633454+0000","last_became_peered":"2023-11-25T05:06:09.633454+0000","last_unstale":"2023-11-25T05:26:14.189467+0000","last_undegraded":"2023-11-25T05:26:14.189467+0000","last_fullsized":"2023-11-25T05:26:14.189467+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:36:34.447428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":863,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.589752+0000","last_change":"2023-11-25T05:05:53.105351+0000","last_active":"2023-11-25T05:26:12.589752+0000","last_peered":"2023-11-25T05:26:12.589752+0000","last_clean":"2023-11-25T05:26:12.589752+0000","last_became_active":"2023-11-25T05:05:53.099253+0000","last_became_peered":"2023-11-25T05:05:53.099253+0000","last_unstale":"2023-11-25T05:26:12.589752+0000","last_undegraded":"2023-11-25T05:26:12.589752+0000","last_fullsized":"2023-11-25T05:26:12.589752+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:25:34.409386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":874,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.210137+0000","last_change":"2023-11-25T05:05:43.502596+0000","last_active":"2023-11-25T05:26:14.210137+0000","last_peered":"2023-11-25T05:26:14.210137+0000","last_clean":"2023-11-25T05:26:14.210137+0000","last_became_active":"2023-11-25T05:05:43.496305+0000","last_became_peered":"2023-11-25T05:05:43.496305+0000","last_unstale":"2023-11-25T05:26:14.210137+0000","last_undegraded":"2023-11-25T05:26:14.210137+0000","last_fullsized":"2023-11-25T05:26:14.210137+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:56:11.421935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":869,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.390854+0000","last_change":"2023-11-25T05:05:42.708461+0000","last_active":"2023-11-25T05:26:11.390854+0000","last_peered":"2023-11-25T05:26:11.390854+0000","last_clean":"2023-11-25T05:26:11.390854+0000","last_became_active":"2023-11-25T05:05:42.672388+0000","last_became_peered":"2023-11-25T05:05:42.672388+0000","last_unstale":"2023-11-25T05:26:11.390854+0000","last_undegraded":"2023-11-25T05:26:11.390854+0000","last_fullsized":"2023-11-25T05:26:11.390854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:34:44.252922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":867,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.460681+0000","last_change":"2023-11-25T05:05:46.966035+0000","last_active":"2023-11-25T05:26:10.460681+0000","last_peered":"2023-11-25T05:26:10.460681+0000","last_clean":"2023-11-25T05:26:10.460681+0000","last_became_active":"2023-11-25T05:05:46.959854+0000","last_became_peered":"2023-11-25T05:05:46.959854+0000","last_unstale":"2023-11-25T05:26:10.460681+0000","last_undegraded":"2023-11-25T05:26:10.460681+0000","last_fullsized":"2023-11-25T05:26:10.460681+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:00:30.622747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":869,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.400561+0000","last_change":"2023-11-25T05:05:47.759205+0000","last_active":"2023-11-25T05:26:11.400561+0000","last_peered":"2023-11-25T05:26:11.400561+0000","last_clean":"2023-11-25T05:26:11.400561+0000","last_became_active":"2023-11-25T05:05:47.753176+0000","last_became_peered":"2023-11-25T05:05:47.753176+0000","last_unstale":"2023-11-25T05:26:11.400561+0000","last_undegraded":"2023-11-25T05:26:11.400561+0000","last_fullsized":"2023-11-25T05:26:11.400561+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:04:21.487913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":870,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.427728+0000","last_change":"2023-11-25T05:05:45.604133+0000","last_active":"2023-11-25T05:26:11.427728+0000","last_peered":"2023-11-25T05:26:11.427728+0000","last_clean":"2023-11-25T05:26:11.427728+0000","last_became_active":"2023-11-25T05:05:45.597733+0000","last_became_peered":"2023-11-25T05:05:45.597733+0000","last_unstale":"2023-11-25T05:26:11.427728+0000","last_undegraded":"2023-11-25T05:26:11.427728+0000","last_fullsized":"2023-11-25T05:26:11.427728+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:20:05.883993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.604334+0000","last_change":"2023-11-25T05:05:47.799603+0000","last_active":"2023-11-25T05:26:13.604334+0000","last_peered":"2023-11-25T05:26:13.604334+0000","last_clean":"2023-11-25T05:26:13.604334+0000","last_became_active":"2023-11-25T05:05:47.792727+0000","last_became_peered":"2023-11-25T05:05:47.792727+0000","last_unstale":"2023-11-25T05:26:13.604334+0000","last_undegraded":"2023-11-25T05:26:13.604334+0000","last_fullsized":"2023-11-25T05:26:13.604334+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:50:46.409556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":861,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.455120+0000","last_change":"2023-11-25T05:04:33.582978+0000","last_active":"2023-11-25T05:26:11.455120+0000","last_peered":"2023-11-25T05:26:11.455120+0000","last_clean":"2023-11-25T05:26:11.455120+0000","last_became_active":"2023-11-25T05:04:33.577510+0000","last_became_peered":"2023-11-25T05:04:33.577510+0000","last_unstale":"2023-11-25T05:26:11.455120+0000","last_undegraded":"2023-11-25T05:26:11.455120+0000","last_fullsized":"2023-11-25T05:26:11.455120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:53:20.091826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":857,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.327012+0000","last_change":"2023-11-25T05:05:58.700776+0000","last_active":"2023-11-25T05:26:10.327012+0000","last_peered":"2023-11-25T05:26:10.327012+0000","last_clean":"2023-11-25T05:26:10.327012+0000","last_became_active":"2023-11-25T05:05:58.695110+0000","last_became_peered":"2023-11-25T05:05:58.695110+0000","last_unstale":"2023-11-25T05:26:10.327012+0000","last_undegraded":"2023-11-25T05:26:10.327012+0000","last_fullsized":"2023-11-25T05:26:10.327012+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T05:50:22.654582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":872,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.591862+0000","last_change":"2023-11-25T05:04:33.645680+0000","last_active":"2023-11-25T05:26:10.591862+0000","last_peered":"2023-11-25T05:26:10.591862+0000","last_clean":"2023-11-25T05:26:10.591862+0000","last_became_active":"2023-11-25T05:04:33.610776+0000","last_became_peered":"2023-11-25T05:04:33.610776+0000","last_unstale":"2023-11-25T05:26:10.591862+0000","last_undegraded":"2023-11-25T05:26:10.591862+0000","last_fullsized":"2023-11-25T05:26:10.591862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:02:08.013900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.680167+0000","last_change":"2023-11-25T05:06:12.667457+0000","last_active":"2023-11-25T05:26:12.680167+0000","last_peered":"2023-11-25T05:26:12.680167+0000","last_clean":"2023-11-25T05:26:12.680167+0000","last_became_active":"2023-11-25T05:06:12.543810+0000","last_became_peered":"2023-11-25T05:06:12.543810+0000","last_unstale":"2023-11-25T05:26:12.680167+0000","last_undegraded":"2023-11-25T05:26:12.680167+0000","last_fullsized":"2023-11-25T05:26:12.680167+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:13:23.439209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":862,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.324782+0000","last_change":"2023-11-25T05:04:33.350255+0000","last_active":"2023-11-25T05:26:10.324782+0000","last_peered":"2023-11-25T05:26:10.324782+0000","last_clean":"2023-11-25T05:26:10.324782+0000","last_became_active":"2023-11-25T05:04:33.337158+0000","last_became_peered":"2023-11-25T05:04:33.337158+0000","last_unstale":"2023-11-25T05:26:10.324782+0000","last_undegraded":"2023-11-25T05:26:10.324782+0000","last_fullsized":"2023-11-25T05:26:10.324782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:25:34.365788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.708724+0000","last_change":"2023-11-25T05:05:51.120038+0000","last_active":"2023-11-25T05:26:12.708724+0000","last_peered":"2023-11-25T05:26:12.708724+0000","last_clean":"2023-11-25T05:26:12.708724+0000","last_became_active":"2023-11-25T05:05:51.097150+0000","last_became_peered":"2023-11-25T05:05:51.097150+0000","last_unstale":"2023-11-25T05:26:12.708724+0000","last_undegraded":"2023-11-25T05:26:12.708724+0000","last_fullsized":"2023-11-25T05:26:12.708724+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:34:02.990987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":873,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.462296+0000","last_change":"2023-11-25T05:04:33.396056+0000","last_active":"2023-11-25T05:26:10.462296+0000","last_peered":"2023-11-25T05:26:10.462296+0000","last_clean":"2023-11-25T05:26:10.462296+0000","last_became_active":"2023-11-25T05:04:33.388849+0000","last_became_peered":"2023-11-25T05:04:33.388849+0000","last_unstale":"2023-11-25T05:26:10.462296+0000","last_undegraded":"2023-11-25T05:26:10.462296+0000","last_fullsized":"2023-11-25T05:26:10.462296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:11:08.763309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":868,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.440673+0000","last_change":"2023-11-25T05:05:55.046489+0000","last_active":"2023-11-25T05:26:10.440673+0000","last_peered":"2023-11-25T05:26:10.440673+0000","last_clean":"2023-11-25T05:26:10.440673+0000","last_became_active":"2023-11-25T05:05:55.038711+0000","last_became_peered":"2023-11-25T05:05:55.038711+0000","last_unstale":"2023-11-25T05:26:10.440673+0000","last_undegraded":"2023-11-25T05:26:10.440673+0000","last_fullsized":"2023-11-25T05:26:10.440673+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:43:58.709876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":862,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.299920+0000","last_change":"2023-11-25T05:04:33.381810+0000","last_active":"2023-11-25T05:26:10.299920+0000","last_peered":"2023-11-25T05:26:10.299920+0000","last_clean":"2023-11-25T05:26:10.299920+0000","last_became_active":"2023-11-25T05:04:33.376375+0000","last_became_peered":"2023-11-25T05:04:33.376375+0000","last_unstale":"2023-11-25T05:26:10.299920+0000","last_undegraded":"2023-11-25T05:26:10.299920+0000","last_fullsized":"2023-11-25T05:26:10.299920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:41:46.038487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":875,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.207787+0000","last_change":"2023-11-25T05:05:44.569896+0000","last_active":"2023-11-25T05:26:14.207787+0000","last_peered":"2023-11-25T05:26:14.207787+0000","last_clean":"2023-11-25T05:26:14.207787+0000","last_became_active":"2023-11-25T05:05:44.554983+0000","last_became_peered":"2023-11-25T05:05:44.554983+0000","last_unstale":"2023-11-25T05:26:14.207787+0000","last_undegraded":"2023-11-25T05:26:14.207787+0000","last_fullsized":"2023-11-25T05:26:14.207787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:59:24.121163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":871,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.564292+0000","last_change":"2023-11-25T05:04:33.238092+0000","last_active":"2023-11-25T05:26:13.564292+0000","last_peered":"2023-11-25T05:26:13.564292+0000","last_clean":"2023-11-25T05:26:13.564292+0000","last_became_active":"2023-11-25T05:04:33.196533+0000","last_became_peered":"2023-11-25T05:04:33.196533+0000","last_unstale":"2023-11-25T05:26:13.564292+0000","last_undegraded":"2023-11-25T05:26:13.564292+0000","last_fullsized":"2023-11-25T05:26:13.564292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:06:22.475692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":868,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.466346+0000","last_change":"2023-11-25T05:05:45.774913+0000","last_active":"2023-11-25T05:26:10.466346+0000","last_peered":"2023-11-25T05:26:10.466346+0000","last_clean":"2023-11-25T05:26:10.466346+0000","last_became_active":"2023-11-25T05:05:45.759824+0000","last_became_peered":"2023-11-25T05:05:45.759824+0000","last_unstale":"2023-11-25T05:26:10.466346+0000","last_undegraded":"2023-11-25T05:26:10.466346+0000","last_fullsized":"2023-11-25T05:26:10.466346+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:41:12.473852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":862,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.249953+0000","last_change":"2023-11-25T05:04:33.294027+0000","last_active":"2023-11-25T05:26:10.249953+0000","last_peered":"2023-11-25T05:26:10.249953+0000","last_clean":"2023-11-25T05:26:10.249953+0000","last_became_active":"2023-11-25T05:04:33.242068+0000","last_became_peered":"2023-11-25T05:04:33.242068+0000","last_unstale":"2023-11-25T05:26:10.249953+0000","last_undegraded":"2023-11-25T05:26:10.249953+0000","last_fullsized":"2023-11-25T05:26:10.249953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:37:54.756624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":868,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.432943+0000","last_change":"2023-11-25T05:05:45.087189+0000","last_active":"2023-11-25T05:26:10.432943+0000","last_peered":"2023-11-25T05:26:10.432943+0000","last_clean":"2023-11-25T05:26:10.432943+0000","last_became_active":"2023-11-25T05:05:45.057546+0000","last_became_peered":"2023-11-25T05:05:45.057546+0000","last_unstale":"2023-11-25T05:26:10.432943+0000","last_undegraded":"2023-11-25T05:26:10.432943+0000","last_fullsized":"2023-11-25T05:26:10.432943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:02:04.146157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":862,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.277063+0000","last_change":"2023-11-25T05:04:33.784328+0000","last_active":"2023-11-25T05:26:10.277063+0000","last_peered":"2023-11-25T05:26:10.277063+0000","last_clean":"2023-11-25T05:26:10.277063+0000","last_became_active":"2023-11-25T05:04:33.778042+0000","last_became_peered":"2023-11-25T05:04:33.778042+0000","last_unstale":"2023-11-25T05:26:10.277063+0000","last_undegraded":"2023-11-25T05:26:10.277063+0000","last_fullsized":"2023-11-25T05:26:10.277063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:40:38.193703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":974,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.192116+0000","last_change":"2023-11-25T05:04:28.241367+0000","last_active":"2023-11-25T05:26:14.192116+0000","last_peered":"2023-11-25T05:26:14.192116+0000","last_clean":"2023-11-25T05:26:14.192116+0000","last_became_active":"2023-11-25T05:04:27.388156+0000","last_became_peered":"2023-11-25T05:04:27.388156+0000","last_unstale":"2023-11-25T05:26:14.192116+0000","last_undegraded":"2023-11-25T05:26:14.192116+0000","last_fullsized":"2023-11-25T05:26:14.192116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:22.310975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:22.310975+0000","last_clean_scrub_stamp":"2023-11-25T05:04:22.310975+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:50:37.122665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":870,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.404755+0000","last_change":"2023-11-25T05:05:47.996637+0000","last_active":"2023-11-25T05:26:11.404755+0000","last_peered":"2023-11-25T05:26:11.404755+0000","last_clean":"2023-11-25T05:26:11.404755+0000","last_became_active":"2023-11-25T05:05:47.973359+0000","last_became_peered":"2023-11-25T05:05:47.973359+0000","last_unstale":"2023-11-25T05:26:11.404755+0000","last_undegraded":"2023-11-25T05:26:11.404755+0000","last_fullsized":"2023-11-25T05:26:11.404755+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:05:25.523743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":848,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-11-25T05:26:09.266673+0000","last_change":"2023-11-25T05:05:54.064947+0000","last_active":"2023-11-25T05:26:09.266673+0000","last_peered":"2023-11-25T05:26:09.266673+0000","last_clean":"2023-11-25T05:26:09.266673+0000","last_became_active":"2023-11-25T05:05:54.045707+0000","last_became_peered":"2023-11-25T05:05:54.045707+0000","last_unstale":"2023-11-25T05:26:09.266673+0000","last_undegraded":"2023-11-25T05:26:09.266673+0000","last_fullsized":"2023-11-25T05:26:09.266673+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T17:02:26.650184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":869,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.428665+0000","last_change":"2023-11-25T05:05:42.506167+0000","last_active":"2023-11-25T05:26:10.428665+0000","last_peered":"2023-11-25T05:26:10.428665+0000","last_clean":"2023-11-25T05:26:10.428665+0000","last_became_active":"2023-11-25T05:05:42.487757+0000","last_became_peered":"2023-11-25T05:05:42.487757+0000","last_unstale":"2023-11-25T05:26:10.428665+0000","last_undegraded":"2023-11-25T05:26:10.428665+0000","last_fullsized":"2023-11-25T05:26:10.428665+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:58:18.951826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"38'7","reported_seq":879,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.562585+0000","last_change":"2023-11-25T05:05:45.233028+0000","last_active":"2023-11-25T05:26:13.562585+0000","last_peered":"2023-11-25T05:26:13.562585+0000","last_clean":"2023-11-25T05:26:13.562585+0000","last_became_active":"2023-11-25T05:05:45.212870+0000","last_became_peered":"2023-11-25T05:05:45.212870+0000","last_unstale":"2023-11-25T05:26:13.562585+0000","last_undegraded":"2023-11-25T05:26:13.562585+0000","last_fullsized":"2023-11-25T05:26:13.562585+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:44:02.468772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.662501+0000","last_change":"2023-11-25T05:05:49.947608+0000","last_active":"2023-11-25T05:26:12.662501+0000","last_peered":"2023-11-25T05:26:12.662501+0000","last_clean":"2023-11-25T05:26:12.662501+0000","last_became_active":"2023-11-25T05:05:49.941593+0000","last_became_peered":"2023-11-25T05:05:49.941593+0000","last_unstale":"2023-11-25T05:26:12.662501+0000","last_undegraded":"2023-11-25T05:26:12.662501+0000","last_fullsized":"2023-11-25T05:26:12.662501+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:02:09.269520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.710445+0000","last_change":"2023-11-25T05:06:26.505899+0000","last_active":"2023-11-25T05:26:12.710445+0000","last_peered":"2023-11-25T05:26:12.710445+0000","last_clean":"2023-11-25T05:26:12.710445+0000","last_became_active":"2023-11-25T05:06:26.500023+0000","last_became_peered":"2023-11-25T05:06:26.500023+0000","last_unstale":"2023-11-25T05:26:12.710445+0000","last_undegraded":"2023-11-25T05:26:12.710445+0000","last_fullsized":"2023-11-25T05:26:12.710445+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T05:42:16.923983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.602721+0000","last_change":"2023-11-25T05:05:48.099018+0000","last_active":"2023-11-25T05:26:13.602721+0000","last_peered":"2023-11-25T05:26:13.602721+0000","last_clean":"2023-11-25T05:26:13.602721+0000","last_became_active":"2023-11-25T05:05:48.093083+0000","last_became_peered":"2023-11-25T05:05:48.093083+0000","last_unstale":"2023-11-25T05:26:13.602721+0000","last_undegraded":"2023-11-25T05:26:13.602721+0000","last_fullsized":"2023-11-25T05:26:13.602721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:36:31.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":867,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.470380+0000","last_change":"2023-11-25T05:05:55.013827+0000","last_active":"2023-11-25T05:26:10.470380+0000","last_peered":"2023-11-25T05:26:10.470380+0000","last_clean":"2023-11-25T05:26:10.470380+0000","last_became_active":"2023-11-25T05:05:55.006070+0000","last_became_peered":"2023-11-25T05:05:55.006070+0000","last_unstale":"2023-11-25T05:26:10.470380+0000","last_undegraded":"2023-11-25T05:26:10.470380+0000","last_fullsized":"2023-11-25T05:26:10.470380+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:42:44.573947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":867,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-11-25T05:26:10.444056+0000","last_change":"2023-11-25T05:05:42.334720+0000","last_active":"2023-11-25T05:26:10.444056+0000","last_peered":"2023-11-25T05:26:10.444056+0000","last_clean":"2023-11-25T05:26:10.444056+0000","last_became_active":"2023-11-25T05:05:42.327418+0000","last_became_peered":"2023-11-25T05:05:42.327418+0000","last_unstale":"2023-11-25T05:26:10.444056+0000","last_undegraded":"2023-11-25T05:26:10.444056+0000","last_fullsized":"2023-11-25T05:26:10.444056+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:45:59.399822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":866,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-11-25T05:26:09.093361+0000","last_change":"2023-11-25T05:05:45.341062+0000","last_active":"2023-11-25T05:26:09.093361+0000","last_peered":"2023-11-25T05:26:09.093361+0000","last_clean":"2023-11-25T05:26:09.093361+0000","last_became_active":"2023-11-25T05:05:45.334024+0000","last_became_peered":"2023-11-25T05:05:45.334024+0000","last_unstale":"2023-11-25T05:26:09.093361+0000","last_undegraded":"2023-11-25T05:26:09.093361+0000","last_fullsized":"2023-11-25T05:26:09.093361+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T05:06:56.822225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.582134+0000","last_change":"2023-11-25T05:06:27.132506+0000","last_active":"2023-11-25T05:26:12.582134+0000","last_peered":"2023-11-25T05:26:12.582134+0000","last_clean":"2023-11-25T05:26:12.582134+0000","last_became_active":"2023-11-25T05:06:27.122861+0000","last_became_peered":"2023-11-25T05:06:27.122861+0000","last_unstale":"2023-11-25T05:26:12.582134+0000","last_undegraded":"2023-11-25T05:26:12.582134+0000","last_fullsized":"2023-11-25T05:26:12.582134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:07:31.032076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":869,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.369980+0000","last_change":"2023-11-25T05:05:57.191530+0000","last_active":"2023-11-25T05:26:11.369980+0000","last_peered":"2023-11-25T05:26:11.369980+0000","last_clean":"2023-11-25T05:26:11.369980+0000","last_became_active":"2023-11-25T05:05:57.177615+0000","last_became_peered":"2023-11-25T05:05:57.177615+0000","last_unstale":"2023-11-25T05:26:11.369980+0000","last_undegraded":"2023-11-25T05:26:11.369980+0000","last_fullsized":"2023-11-25T05:26:11.369980+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:21:49.038926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":16,"seq":68719476999,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66640,"kb_used_data":13244,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305200,"statfs":{"total":96636764160,"available":96568524800,"internally_reserved":0,"allocated":13561856,"data_stored":8262098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":60,"apply_latency_ms":60,"commit_latency_ns":60000000,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":17.164000000000001,"5min":25.925000000000001,"15min":25.350999999999999},"min":{"1min":4.9020000000000001,"5min":4.9020000000000001,"15min":4.9020000000000001},"max":{"1min":100.782,"5min":164.57300000000001,"15min":203.28800000000001},"last":10.269},{"interface":"front","average":{"1min":15.237,"5min":26.946000000000002,"15min":26.018000000000001},"min":{"1min":5.7000000000000002,"5min":5.6859999999999999,"15min":5.4400000000000004},"max":{"1min":92.989000000000004,"5min":238.51300000000001,"15min":347.452},"last":17.202000000000002}]},{"osd":1,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":20.888000000000002,"5min":26.521999999999998,"15min":25.02},"min":{"1min":8.3249999999999993,"5min":7.4660000000000002,"15min":5.6589999999999998},"max":{"1min":86.962999999999994,"5min":287.27199999999999,"15min":287.27199999999999},"last":19.829999999999998},{"interface":"front","average":{"1min":17.350999999999999,"5min":27.222000000000001,"15min":23.896000000000001},"min":{"1min":5.282,"5min":5.282,"15min":5.282},"max":{"1min":57.453000000000003,"5min":286.57400000000001,"15min":286.57400000000001},"last":11.914}]},{"osd":3,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":23.404,"5min":20.637,"15min":20.605},"min":{"1min":6.3289999999999997,"5min":6.3289999999999997,"15min":5.9459999999999997},"max":{"1min":167.83799999999999,"5min":167.83799999999999,"15min":177.66},"last":9.5030000000000001},{"interface":"front","average":{"1min":30.977,"5min":28.187999999999999,"15min":31.155999999999999},"min":{"1min":11.647,"5min":8.1829999999999998,"15min":7.609},"max":{"1min":156.17699999999999,"5min":156.17699999999999,"15min":273.99900000000002},"last":15.653}]},{"osd":4,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":20.108000000000001,"5min":29.747,"15min":26.818000000000001},"min":{"1min":7.7789999999999999,"5min":6.9109999999999996,"15min":6.9109999999999996},"max":{"1min":71.611000000000004,"5min":275.66300000000001,"15min":349.90300000000002},"last":12.734999999999999},{"interface":"front","average":{"1min":17.427,"5min":26.164999999999999,"15min":22.431000000000001},"min":{"1min":7.1589999999999998,"5min":7.1589999999999998,"15min":6.6760000000000002},"max":{"1min":73.219999999999999,"5min":285.12700000000001,"15min":285.12700000000001},"last":7.8719999999999999}]},{"osd":5,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":22.722999999999999,"5min":24.5,"15min":29.382999999999999},"min":{"1min":11.710000000000001,"5min":9.7840000000000007,"15min":8.0619999999999994},"max":{"1min":89.492000000000004,"5min":138.34899999999999,"15min":353.63400000000001},"last":15.005000000000001},{"interface":"front","average":{"1min":26.366,"5min":30.738,"15min":34.677999999999997},"min":{"1min":14.381,"5min":13.541,"15min":11.539999999999999},"max":{"1min":96.620999999999995,"5min":146.41300000000001,"15min":354.56200000000001},"last":21.600999999999999}]},{"osd":6,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":31.605,"5min":30.536999999999999,"15min":31.172000000000001},"min":{"1min":16.321000000000002,"5min":11.292999999999999,"15min":9.7590000000000003},"max":{"1min":153.01400000000001,"5min":153.01400000000001,"15min":213.11000000000001},"last":20.943000000000001},{"interface":"front","average":{"1min":33.220999999999997,"5min":34.654000000000003,"15min":34.892000000000003},"min":{"1min":15.217000000000001,"5min":12.462999999999999,"15min":12.462999999999999},"max":{"1min":144.51400000000001,"5min":288.30000000000001,"15min":288.30000000000001},"last":22.247}]},{"osd":7,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":25.690999999999999,"5min":27.300000000000001,"15min":26.981000000000002},"min":{"1min":11.557,"5min":6.375,"15min":6.375},"max":{"1min":90.355999999999995,"5min":216.095,"15min":350.81599999999997},"last":17.853999999999999},{"interface":"front","average":{"1min":21.873000000000001,"5min":22.734000000000002,"15min":23.170000000000002},"min":{"1min":8.6400000000000006,"5min":7.5419999999999998,"15min":7.1669999999999998},"max":{"1min":96.100999999999999,"5min":141.59100000000001,"15min":210.55500000000001},"last":13.403}]}]},{"osd":1,"up_from":16,"seq":68719476999,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67056,"kb_used_data":13648,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94304784,"statfs":{"total":96636764160,"available":96568098816,"internally_reserved":0,"allocated":13975552,"data_stored":8666539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":23.388000000000002,"5min":20.16,"15min":21.632000000000001},"min":{"1min":6.165,"5min":5.9809999999999999,"15min":5.1600000000000001},"max":{"1min":163.62,"5min":221.589,"15min":349.75799999999998},"last":35.331000000000003},{"interface":"front","average":{"1min":32.435000000000002,"5min":29.16,"15min":26.466999999999999},"min":{"1min":9.4299999999999997,"5min":7.1529999999999996,"15min":6.5309999999999997},"max":{"1min":240.31200000000001,"5min":318.02600000000001,"15min":318.02600000000001},"last":28.692}]},{"osd":2,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":28.914000000000001,"5min":20.263000000000002,"15min":22.899999999999999},"min":{"1min":6.7889999999999997,"5min":5.1429999999999998,"15min":4.9619999999999997},"max":{"1min":148.28299999999999,"5min":161.10900000000001,"15min":353.286},"last":17.966000000000001},{"interface":"front","average":{"1min":34.442,"5min":21.100999999999999,"15min":19.414999999999999},"min":{"1min":7.7750000000000004,"5min":6.3280000000000003,"15min":5.7149999999999999},"max":{"1min":169.84100000000001,"5min":169.84100000000001,"15min":247.99199999999999},"last":30.224}]},{"osd":3,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":30.649000000000001,"5min":28.231999999999999,"15min":24.68},"min":{"1min":7.0339999999999998,"5min":6.665,"15min":6.5899999999999999},"max":{"1min":169.172,"5min":244.58199999999999,"15min":440.05599999999998},"last":33.240000000000002},{"interface":"front","average":{"1min":31.716000000000001,"5min":30.024000000000001,"15min":28.202999999999999},"min":{"1min":8.4369999999999994,"5min":8.4369999999999994,"15min":5.9889999999999999},"max":{"1min":168.58000000000001,"5min":242.29400000000001,"15min":383.39800000000002},"last":28.02}]},{"osd":4,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":20.422000000000001,"5min":17.603000000000002,"15min":21.190999999999999},"min":{"1min":8.9909999999999997,"5min":6.7830000000000004,"15min":6.5789999999999997},"max":{"1min":63.802999999999997,"5min":113.84,"15min":360.161},"last":34.058},{"interface":"front","average":{"1min":27.309999999999999,"5min":24.93,"15min":23.541},"min":{"1min":9.0510000000000002,"5min":9.0510000000000002,"15min":7.4409999999999998},"max":{"1min":114.33199999999999,"5min":226.68000000000001,"15min":364.39800000000002},"last":37.947000000000003}]},{"osd":5,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":39.484999999999999,"5min":28.780000000000001,"15min":29.396999999999998},"min":{"1min":9.6669999999999998,"5min":9.6669999999999998,"15min":7.4359999999999999},"max":{"1min":157.357,"5min":157.357,"15min":282.63900000000001},"last":99.200999999999993},{"interface":"front","average":{"1min":29.167999999999999,"5min":24.120999999999999,"15min":26.896000000000001},"min":{"1min":7.6630000000000003,"5min":7.2889999999999997,"15min":7.1529999999999996},"max":{"1min":105.211,"5min":163.33600000000001,"15min":382.01400000000001},"last":126.55}]},{"osd":6,"last update":"Sat Nov 25 05:25:21 2023","interfaces":[{"interface":"back","average":{"1min":18.364000000000001,"5min":24.588000000000001,"15min":47.482999999999997},"min":{"1min":10.593999999999999,"5min":7.5209999999999999,"15min":6.8230000000000004},"max":{"1min":64.594999999999999,"5min":187.62100000000001,"15min":3420.1689999999999},"last":32.476999999999997},{"interface":"front","average":{"1min":22.228999999999999,"5min":26.702000000000002,"15min":40.634999999999998},"min":{"1min":12.193,"5min":9.9260000000000002,"15min":9.423},"max":{"1min":70.814999999999998,"5min":131.99600000000001,"15min":1922.326},"last":34.700000000000003}]},{"osd":7,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":23.481999999999999,"5min":23.608000000000001,"15min":24.239999999999998},"min":{"1min":8.8559999999999999,"5min":7.0279999999999996,"15min":7.0279999999999996},"max":{"1min":116.39400000000001,"5min":219.57900000000001,"15min":272.03699999999998},"last":29.381},{"interface":"front","average":{"1min":30.045999999999999,"5min":26.843,"15min":27.969000000000001},"min":{"1min":11.65,"5min":10.239000000000001,"15min":8.3889999999999993},"max":{"1min":118.447,"5min":126.80200000000001,"15min":297.976},"last":35.962000000000003}]}]},{"osd":3,"up_from":16,"seq":68719476999,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84772,"kb_used_data":13304,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94287068,"statfs":{"total":96636764160,"available":96549957632,"internally_reserved":0,"allocated":13623296,"data_stored":8341244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":21.867000000000001,"5min":16.334,"15min":21.956},"min":{"1min":5.556,"5min":4.3090000000000002,"15min":4.3090000000000002},"max":{"1min":100.45399999999999,"5min":124.46299999999999,"15min":394.84199999999998},"last":36.960999999999999},{"interface":"front","average":{"1min":19.355,"5min":17.161000000000001,"15min":21.390999999999998},"min":{"1min":7.2969999999999997,"5min":4.9020000000000001,"15min":4.5890000000000004},"max":{"1min":84.197000000000003,"5min":119.611,"15min":379.51900000000001},"last":37.802}]},{"osd":1,"last update":"Sat Nov 25 05:25:12 2023","interfaces":[{"interface":"back","average":{"1min":16.382000000000001,"5min":19.366,"15min":37.366},"min":{"1min":7.2450000000000001,"5min":7.1470000000000002,"15min":5.4080000000000004},"max":{"1min":74.954999999999998,"5min":132.477,"15min":674.39599999999996},"last":11.731},{"interface":"front","average":{"1min":19.981999999999999,"5min":24.111000000000001,"15min":34.749000000000002},"min":{"1min":9.0239999999999991,"5min":8.1419999999999995,"15min":7.2729999999999997},"max":{"1min":77.009,"5min":253.03100000000001,"15min":454.755},"last":8.8369999999999997}]},{"osd":2,"last update":"Sat Nov 25 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":18.132000000000001,"5min":16.486999999999998,"15min":34.378999999999998},"min":{"1min":7.798,"5min":7.2649999999999997,"15min":7.0620000000000003},"max":{"1min":83.954999999999998,"5min":83.954999999999998,"15min":673.12400000000002},"last":40.529000000000003},{"interface":"front","average":{"1min":22.617000000000001,"5min":22.503,"15min":35.618000000000002},"min":{"1min":7.351,"5min":7.351,"15min":7.351},"max":{"1min":77.671999999999997,"5min":227.36199999999999,"15min":492.04899999999998},"last":39.125}]},{"osd":4,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":21.428999999999998,"5min":21.751999999999999,"15min":31.024000000000001},"min":{"1min":8.8360000000000003,"5min":8.8360000000000003,"15min":7.6890000000000001},"max":{"1min":81.578000000000003,"5min":141.38399999999999,"15min":676.21299999999997},"last":16.596},{"interface":"front","average":{"1min":19.236000000000001,"5min":20.013999999999999,"15min":23.495999999999999},"min":{"1min":7.6040000000000001,"5min":6.641,"15min":6.1559999999999997},"max":{"1min":77.034999999999997,"5min":313.18299999999999,"15min":313.18299999999999},"last":12.586}]},{"osd":5,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":26.818000000000001,"5min":23.085999999999999,"15min":32.587000000000003},"min":{"1min":10.710000000000001,"5min":9.6999999999999993,"15min":9.3360000000000003},"max":{"1min":80.661000000000001,"5min":194.51400000000001,"15min":675.37800000000004},"last":15.000999999999999},{"interface":"front","average":{"1min":27.657,"5min":21.199999999999999,"15min":28.454999999999998},"min":{"1min":11.975,"5min":7.5190000000000001,"15min":7.5190000000000001},"max":{"1min":103.021,"5min":131.768,"15min":314.40800000000002},"last":29.794}]},{"osd":6,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":23.952999999999999,"5min":20.739999999999998,"15min":24.882000000000001},"min":{"1min":8.5519999999999996,"5min":8.3030000000000008,"15min":7.0190000000000001},"max":{"1min":83.287999999999997,"5min":172.66,"15min":312.20699999999999},"last":15.878},{"interface":"front","average":{"1min":26.363,"5min":24.664000000000001,"15min":35.289999999999999},"min":{"1min":11.327,"5min":10.263999999999999,"15min":10.029},"max":{"1min":102.366,"5min":185.809,"15min":677.03499999999997},"last":18.41}]},{"osd":7,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":23.309000000000001,"5min":23.741,"15min":28.478999999999999},"min":{"1min":11.162000000000001,"5min":8.1609999999999996,"15min":7.6420000000000003},"max":{"1min":68.536000000000001,"5min":315.61500000000001,"15min":315.61500000000001},"last":9.6869999999999994},{"interface":"front","average":{"1min":26.361999999999998,"5min":24.396000000000001,"15min":30.718},"min":{"1min":10.535,"5min":10.535,"15min":10.449999999999999},"max":{"1min":102.361,"5min":145.86099999999999,"15min":320.38400000000001},"last":10.839}]}]},{"osd":0,"up_from":16,"seq":68719477000,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66664,"kb_used_data":13268,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305176,"statfs":{"total":96636764160,"available":96568500224,"internally_reserved":0,"allocated":13586432,"data_stored":8308714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":12.904999999999999,"5min":15.397,"15min":17.724},"min":{"1min":4.6100000000000003,"5min":4.6100000000000003,"15min":4.6100000000000003},"max":{"1min":39.119,"5min":109.95399999999999,"15min":209.29900000000001},"last":23.010999999999999},{"interface":"front","average":{"1min":12.183999999999999,"5min":16.882000000000001,"15min":19.984999999999999},"min":{"1min":5.6219999999999999,"5min":4.5869999999999997,"15min":4.4420000000000002},"max":{"1min":56.362000000000002,"5min":119.73,"15min":401.36000000000001},"last":16.911000000000001}]},{"osd":2,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":17.683,"5min":19.295000000000002,"15min":22.402999999999999},"min":{"1min":6.6680000000000001,"5min":6.5380000000000003,"15min":6.1260000000000003},"max":{"1min":71.555999999999997,"5min":94.311999999999998,"15min":460.572},"last":35.844999999999999},{"interface":"front","average":{"1min":17.940000000000001,"5min":18.923999999999999,"15min":22.27},"min":{"1min":7.4980000000000002,"5min":6.5279999999999996,"15min":6.3090000000000002},"max":{"1min":73.290000000000006,"5min":89.322999999999993,"15min":405.94099999999997},"last":37.648000000000003}]},{"osd":3,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":21.096,"5min":23.585999999999999,"15min":24.260000000000002},"min":{"1min":7.2999999999999998,"5min":7.2999999999999998,"15min":6.8540000000000001},"max":{"1min":53.616999999999997,"5min":157.65299999999999,"15min":267.74599999999998},"last":29.195},{"interface":"front","average":{"1min":19.989000000000001,"5min":23.132000000000001,"15min":23.312000000000001},"min":{"1min":9.8480000000000008,"5min":7.1269999999999998,"15min":6.4080000000000004},"max":{"1min":55.122,"5min":128.14500000000001,"15min":266.81900000000002},"last":24.175999999999998}]},{"osd":4,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":21.091000000000001,"5min":20.103999999999999,"15min":20.606000000000002},"min":{"1min":9.4649999999999999,"5min":8.3940000000000001,"15min":6.2750000000000004},"max":{"1min":72.540999999999997,"5min":115.369,"15min":147.86099999999999},"last":11.446},{"interface":"front","average":{"1min":26.452000000000002,"5min":23.149999999999999,"15min":22.498999999999999},"min":{"1min":9.2189999999999994,"5min":6.8250000000000002,"15min":6.8250000000000002},"max":{"1min":110.93300000000001,"5min":127.291,"15min":148.96199999999999},"last":9.4199999999999999}]},{"osd":5,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.902999999999999,"5min":23.966999999999999,"15min":23.213999999999999},"min":{"1min":9.0109999999999992,"5min":6.9829999999999997,"15min":6.9829999999999997},"max":{"1min":69.957999999999998,"5min":184.80000000000001,"15min":184.80000000000001},"last":136},{"interface":"front","average":{"1min":20.024000000000001,"5min":23.408999999999999,"15min":25.248000000000001},"min":{"1min":10.836,"5min":7.3849999999999998,"15min":7.3849999999999998},"max":{"1min":58.210000000000001,"5min":131.583,"15min":395.65100000000001},"last":227.119}]},{"osd":6,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":21.329000000000001,"5min":23.550000000000001,"15min":24.617000000000001},"min":{"1min":9.2829999999999995,"5min":8.4809999999999999,"15min":8.4809999999999999},"max":{"1min":71.225999999999999,"5min":116.63800000000001,"15min":161.24299999999999},"last":18.061},{"interface":"front","average":{"1min":26.015999999999998,"5min":25.968,"15min":26.870000000000001},"min":{"1min":13.154999999999999,"5min":9.7759999999999998,"15min":9.7759999999999998},"max":{"1min":71.831999999999994,"5min":115.986,"15min":138.46799999999999},"last":15.566000000000001}]},{"osd":7,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.158000000000001,"5min":18.911000000000001,"15min":20.722999999999999},"min":{"1min":10.941000000000001,"5min":8.1340000000000003,"15min":7.1079999999999997},"max":{"1min":40.393999999999998,"5min":112.73099999999999,"15min":140.595},"last":21.469999999999999},{"interface":"front","average":{"1min":21.975999999999999,"5min":22.829999999999998,"15min":25.202999999999999},"min":{"1min":11.441000000000001,"5min":10.888999999999999,"15min":9.9860000000000007},"max":{"1min":57.088000000000001,"5min":103.19,"15min":339.298},"last":42.442999999999998}]}]},{"osd":6,"up_from":15,"seq":64424509704,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84768,"kb_used_data":13312,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94287072,"statfs":{"total":96636764160,"available":96549961728,"internally_reserved":0,"allocated":13631488,"data_stored":8340674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":16.198,"5min":17.219999999999999,"15min":19.709},"min":{"1min":4.7489999999999997,"5min":4.6740000000000004,"15min":4.4409999999999998},"max":{"1min":117.05500000000001,"5min":117.05500000000001,"15min":203.857},"last":25.263000000000002},{"interface":"front","average":{"1min":15.382999999999999,"5min":16.821000000000002,"15min":20.439},"min":{"1min":5.4820000000000002,"5min":5.1210000000000004,"15min":4.6289999999999996},"max":{"1min":117.795,"5min":236.78700000000001,"15min":381.43900000000002},"last":31.957999999999998}]},{"osd":1,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.943999999999999,"5min":22.879000000000001,"15min":26.055},"min":{"1min":8.0890000000000004,"5min":6.2839999999999998,"15min":6.165},"max":{"1min":143.61099999999999,"5min":229.53399999999999,"15min":310.423},"last":11.465999999999999},{"interface":"front","average":{"1min":20.885999999999999,"5min":20.399999999999999,"15min":26.382999999999999},"min":{"1min":6.3360000000000003,"5min":6.2030000000000003,"15min":5.1310000000000002},"max":{"1min":190.029,"5min":243.74700000000001,"15min":548.03599999999994},"last":5.9429999999999996}]},{"osd":2,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":28.033999999999999,"5min":22.832000000000001,"15min":27.013999999999999},"min":{"1min":7.9550000000000001,"5min":6.9690000000000003,"15min":6.9690000000000003},"max":{"1min":122.754,"5min":251.524,"15min":353.80399999999997},"last":90.819000000000003},{"interface":"front","average":{"1min":26.027000000000001,"5min":27.611999999999998,"15min":30.841999999999999},"min":{"1min":10.65,"5min":10.65,"15min":10.617000000000001},"max":{"1min":120.961,"5min":249.97399999999999,"15min":304.49900000000002},"last":90.048000000000002}]},{"osd":3,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":25.802,"5min":31.065999999999999,"15min":32.701999999999998},"min":{"1min":11.24,"5min":10.164,"15min":7.5419999999999998},"max":{"1min":185.28299999999999,"5min":259.89299999999997,"15min":295.82799999999997},"last":156.45099999999999},{"interface":"front","average":{"1min":22.247,"5min":26.661000000000001,"15min":26.797999999999998},"min":{"1min":7.7720000000000002,"5min":6.9199999999999999,"15min":6.0209999999999999},"max":{"1min":168.56100000000001,"5min":249.07400000000001,"15min":266.40199999999999},"last":181.51599999999999}]},{"osd":4,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":31.027999999999999,"5min":31.033000000000001,"15min":32.439999999999998},"min":{"1min":10.000999999999999,"5min":10.000999999999999,"15min":10.000999999999999},"max":{"1min":186.28,"5min":277.85000000000002,"15min":330.86900000000003},"last":12.423},{"interface":"front","average":{"1min":25.614999999999998,"5min":26.050000000000001,"15min":25.638000000000002},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":6.5919999999999996},"max":{"1min":192.25999999999999,"5min":288.13099999999997,"15min":288.13099999999997},"last":8.3740000000000006}]},{"osd":5,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.140000000000001,"5min":18.035,"15min":22.939},"min":{"1min":6.0039999999999996,"5min":5.5590000000000002,"15min":5.5590000000000002},"max":{"1min":121.31399999999999,"5min":237.46100000000001,"15min":281.13999999999999},"last":7.5819999999999999},{"interface":"front","average":{"1min":25.876000000000001,"5min":28.905999999999999,"15min":33.758000000000003},"min":{"1min":8.0069999999999997,"5min":8.0069999999999997,"15min":8.0069999999999997},"max":{"1min":133.44999999999999,"5min":269.77100000000002,"15min":311.09199999999998},"last":13.314}]},{"osd":7,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":29.199999999999999,"5min":31.532,"15min":34.887999999999998},"min":{"1min":13.699,"5min":12.319000000000001,"15min":9.2210000000000001},"max":{"1min":140.25200000000001,"5min":259.87700000000001,"15min":367.24900000000002},"last":14.119},{"interface":"front","average":{"1min":25.48,"5min":24.911999999999999,"15min":28.138999999999999},"min":{"1min":8.8529999999999998,"5min":8.3469999999999995,"15min":7.7199999999999998},"max":{"1min":161.37899999999999,"5min":269.14400000000001,"15min":325.54199999999997},"last":9.8339999999999996}]}]},{"osd":4,"up_from":15,"seq":64424509705,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70828,"kb_used_data":13260,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94301012,"statfs":{"total":96636764160,"available":96564236288,"internally_reserved":0,"allocated":13578240,"data_stored":8296431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":17.417999999999999,"5min":18.896000000000001,"15min":24.765999999999998},"min":{"1min":5.0439999999999996,"5min":4.3170000000000002,"15min":3.8279999999999998},"max":{"1min":103.83199999999999,"5min":150.73599999999999,"15min":361.67000000000002},"last":5.7460000000000004},{"interface":"front","average":{"1min":12.904,"5min":17.263999999999999,"15min":23.478999999999999},"min":{"1min":5.3449999999999998,"5min":4.5369999999999999,"15min":4.1289999999999996},"max":{"1min":55.915999999999997,"5min":216.86600000000001,"15min":270.81},"last":9.6690000000000005}]},{"osd":1,"last update":"Sat Nov 25 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":12.121,"5min":19.879999999999999,"15min":24.692},"min":{"1min":5.2149999999999999,"5min":5.0519999999999996,"15min":4.9749999999999996},"max":{"1min":58.972000000000001,"5min":205.36500000000001,"15min":445.61500000000001},"last":10.574999999999999},{"interface":"front","average":{"1min":15.630000000000001,"5min":20.486000000000001,"15min":24.024000000000001},"min":{"1min":10.656000000000001,"5min":6.5759999999999996,"15min":6.5759999999999996},"max":{"1min":30.831,"5min":155.90299999999999,"15min":274.81099999999998},"last":9.2379999999999995}]},{"osd":2,"last update":"Sat Nov 25 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":15.132,"5min":20.442,"15min":28.692},"min":{"1min":9.8219999999999992,"5min":6.9320000000000004,"15min":6.9320000000000004},"max":{"1min":31.68,"5min":170.72200000000001,"15min":334.29399999999998},"last":13.242000000000001},{"interface":"front","average":{"1min":21.068000000000001,"5min":24.486000000000001,"15min":31.969999999999999},"min":{"1min":11.401,"5min":10.664,"15min":8.5899999999999999},"max":{"1min":74.477000000000004,"5min":165.952,"15min":348.54599999999999},"last":14.807}]},{"osd":3,"last update":"Sat Nov 25 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":14.75,"5min":24.774000000000001,"15min":37.012},"min":{"1min":8.0649999999999995,"5min":6.5810000000000004,"15min":6.133},"max":{"1min":40.886000000000003,"5min":206.09899999999999,"15min":588.27800000000002},"last":11.724},{"interface":"front","average":{"1min":16.039000000000001,"5min":25.753,"15min":28.155000000000001},"min":{"1min":6.9210000000000003,"5min":6.4740000000000002,"15min":6.4740000000000002},"max":{"1min":75.966999999999999,"5min":204.739,"15min":381.53100000000001},"last":15.718999999999999}]},{"osd":5,"last update":"Sat Nov 25 05:25:56 2023","interfaces":[{"interface":"back","average":{"1min":18.43,"5min":20.623999999999999,"15min":26.562000000000001},"min":{"1min":5.875,"5min":5.875,"15min":5.875},"max":{"1min":72.236999999999995,"5min":195.851,"15min":360.387},"last":16.986999999999998},{"interface":"front","average":{"1min":24.617000000000001,"5min":24.105,"15min":29.986999999999998},"min":{"1min":12.101000000000001,"5min":9.4469999999999992,"15min":9.4469999999999992},"max":{"1min":122.352,"5min":159.67099999999999,"15min":274.38200000000001},"last":16.393000000000001}]},{"osd":6,"last update":"Sat Nov 25 05:25:56 2023","interfaces":[{"interface":"back","average":{"1min":20.521000000000001,"5min":24.015000000000001,"15min":27.704999999999998},"min":{"1min":10.093,"5min":10.093,"15min":9.8409999999999993},"max":{"1min":48.161000000000001,"5min":169.44499999999999,"15min":181.43299999999999},"last":17.602},{"interface":"front","average":{"1min":24.329000000000001,"5min":22.428999999999998,"15min":23.622},"min":{"1min":15.603999999999999,"5min":7.1619999999999999,"15min":5.6390000000000002},"max":{"1min":60.040999999999997,"5min":212.25399999999999,"15min":219.26499999999999},"last":18.332000000000001}]},{"osd":7,"last update":"Sat Nov 25 05:25:56 2023","interfaces":[{"interface":"back","average":{"1min":14.73,"5min":23.181000000000001,"15min":25.620999999999999},"min":{"1min":8.7100000000000009,"5min":7.0369999999999999,"15min":7.0369999999999999},"max":{"1min":27.728999999999999,"5min":202.363,"15min":215.512},"last":19.777999999999999},{"interface":"front","average":{"1min":14.417999999999999,"5min":20.943999999999999,"15min":24.698},"min":{"1min":6.0350000000000001,"5min":6.0350000000000001,"15min":6.0350000000000001},"max":{"1min":29.920999999999999,"5min":211.62200000000001,"15min":248.83199999999999},"last":12.367000000000001}]}]},{"osd":7,"up_from":13,"seq":55834575112,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66668,"kb_used_data":13272,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305172,"statfs":{"total":96636764160,"available":96568496128,"internally_reserved":0,"allocated":13590528,"data_stored":8295623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":11.608000000000001,"5min":13.819000000000001,"15min":17.303999999999998},"min":{"1min":4.6150000000000002,"5min":4.3579999999999997,"15min":4.3579999999999997},"max":{"1min":23.535,"5min":97.569000000000003,"15min":222.50800000000001},"last":28.766999999999999},{"interface":"front","average":{"1min":9.8520000000000003,"5min":15.375999999999999,"15min":18.193999999999999},"min":{"1min":4.7450000000000001,"5min":4.6600000000000001,"15min":4.5579999999999998},"max":{"1min":25.248000000000001,"5min":218.03999999999999,"15min":218.03999999999999},"last":34.869999999999997}]},{"osd":1,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.643000000000001,"5min":13.715999999999999,"15min":17.489000000000001},"min":{"1min":5.4859999999999998,"5min":5.4859999999999998,"15min":5.2619999999999996},"max":{"1min":79.078999999999994,"5min":79.078999999999994,"15min":165.99000000000001},"last":8.4079999999999995},{"interface":"front","average":{"1min":20.498000000000001,"5min":18.827000000000002,"15min":22.771999999999998},"min":{"1min":7.508,"5min":6.8140000000000001,"15min":6.7030000000000003},"max":{"1min":117.25,"5min":117.25,"15min":303.892},"last":10.375}]},{"osd":2,"last update":"Sat Nov 25 05:25:16 2023","interfaces":[{"interface":"back","average":{"1min":16.558,"5min":20.265000000000001,"15min":24.670999999999999},"min":{"1min":8.1530000000000005,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":26.408000000000001,"5min":186.08600000000001,"15min":214.37100000000001},"last":21.869},{"interface":"front","average":{"1min":13.426,"5min":19.419,"15min":23.486999999999998},"min":{"1min":7.8540000000000001,"5min":7.508,"15min":6.4279999999999999},"max":{"1min":23.759,"5min":192.584,"15min":337.41899999999998},"last":22.702000000000002}]},{"osd":3,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":16.559999999999999,"5min":22.437999999999999,"15min":23.844999999999999},"min":{"1min":12.122999999999999,"5min":8.4469999999999992,"15min":7.5199999999999996},"max":{"1min":28.463000000000001,"5min":152.47399999999999,"15min":238.941},"last":13.661},{"interface":"front","average":{"1min":18.123000000000001,"5min":21.559999999999999,"15min":26.684999999999999},"min":{"1min":12.221,"5min":6.6929999999999996,"15min":5.5949999999999998},"max":{"1min":29.536000000000001,"5min":150.208,"15min":1292.413},"last":17.611999999999998}]},{"osd":4,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.506,"5min":23.238,"15min":25.268000000000001},"min":{"1min":8.8109999999999999,"5min":8.8109999999999999,"15min":7.9340000000000002},"max":{"1min":21.449999999999999,"5min":120.414,"15min":328.31400000000002},"last":10.839},{"interface":"front","average":{"1min":27.879000000000001,"5min":21.405000000000001,"15min":23.079999999999998},"min":{"1min":10.739000000000001,"5min":7.1849999999999996,"15min":6.4320000000000004},"max":{"1min":192.684,"5min":192.684,"15min":302.97199999999998},"last":27.952000000000002}]},{"osd":5,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":17.332000000000001,"5min":22.279,"15min":24.291},"min":{"1min":9.1509999999999998,"5min":7.7919999999999998,"15min":7.7619999999999996},"max":{"1min":42.118000000000002,"5min":186.69499999999999,"15min":217.023},"last":21.094000000000001},{"interface":"front","average":{"1min":23.495000000000001,"5min":28.128,"15min":29.045999999999999},"min":{"1min":11.694000000000001,"5min":10.643000000000001,"15min":7.1130000000000004},"max":{"1min":83.798000000000002,"5min":210.22300000000001,"15min":318.11599999999999},"last":30.222000000000001}]},{"osd":6,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.75,"5min":20.798999999999999,"15min":22.933},"min":{"1min":7.069,"5min":6.2380000000000004,"15min":6.2380000000000004},"max":{"1min":100.828,"5min":188.15199999999999,"15min":282.28100000000001},"last":26.225999999999999},{"interface":"front","average":{"1min":18.027999999999999,"5min":23.329000000000001,"15min":25.382000000000001},"min":{"1min":8.0459999999999994,"5min":8.0459999999999994,"15min":8.0459999999999994},"max":{"1min":100.143,"5min":187.09899999999999,"15min":215.00700000000001},"last":47.826000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575111,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89260,"kb_used_data":13644,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94282580,"statfs":{"total":96636764160,"available":96545361920,"internally_reserved":0,"allocated":13971456,"data_stored":8685836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":64,"apply_latency_ms":64,"commit_latency_ns":64000000,"apply_latency_ns":64000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":17.623000000000001,"5min":14.541,"15min":18.605},"min":{"1min":4.5910000000000002,"5min":4.202,"15min":4.202},"max":{"1min":133.92599999999999,"5min":167.83600000000001,"15min":471.81},"last":9.4380000000000006},{"interface":"front","average":{"1min":14.865,"5min":15.755000000000001,"15min":20.606999999999999},"min":{"1min":6.2699999999999996,"5min":5.4279999999999999,"15min":4.5529999999999999},"max":{"1min":46.905000000000001,"5min":197.834,"15min":317.45699999999999},"last":7.492}]},{"osd":1,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":27.741,"5min":18.356000000000002,"15min":21.565000000000001},"min":{"1min":6.3470000000000004,"5min":5.4189999999999996,"15min":5.0709999999999997},"max":{"1min":140.28999999999999,"5min":140.28999999999999,"15min":263.589},"last":8.4860000000000007},{"interface":"front","average":{"1min":30.024000000000001,"5min":17.751000000000001,"15min":23.059999999999999},"min":{"1min":6.351,"5min":5.1459999999999999,"15min":5.1459999999999999},"max":{"1min":150.196,"5min":150.196,"15min":238.26499999999999},"last":7.173}]},{"osd":2,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":19.062999999999999,"5min":19.102,"15min":23.210000000000001},"min":{"1min":7.1890000000000001,"5min":6.2240000000000002,"15min":6.2240000000000002},"max":{"1min":82.295000000000002,"5min":182.477,"15min":416.83199999999999},"last":12.640000000000001},{"interface":"front","average":{"1min":28.707000000000001,"5min":25.138000000000002,"15min":25.091999999999999},"min":{"1min":12.006,"5min":9.4000000000000004,"15min":7.0880000000000001},"max":{"1min":179.749,"5min":190.517,"15min":207.28700000000001},"last":12.006}]},{"osd":3,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":31.044,"5min":24.114000000000001,"15min":29.751999999999999},"min":{"1min":10.327,"5min":10.097,"15min":6.6079999999999997},"max":{"1min":166.59700000000001,"5min":166.59700000000001,"15min":443.10300000000001},"last":13.262},{"interface":"front","average":{"1min":28.024999999999999,"5min":19.308,"15min":31.902000000000001},"min":{"1min":7.851,"5min":6.9800000000000004,"15min":6.2199999999999998},"max":{"1min":130.87299999999999,"5min":130.87299999999999,"15min":1666.6489999999999},"last":16.448}]},{"osd":4,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":30.039000000000001,"5min":25.219999999999999,"15min":25.515999999999998},"min":{"1min":10.957000000000001,"5min":10.657,"15min":9.0280000000000005},"max":{"1min":173.041,"5min":173.041,"15min":173.041},"last":14.476000000000001},{"interface":"front","average":{"1min":17.225999999999999,"5min":17.971,"15min":21.449000000000002},"min":{"1min":6.5330000000000004,"5min":6.5330000000000004,"15min":6.5330000000000004},"max":{"1min":68.790000000000006,"5min":123.01300000000001,"15min":156.92500000000001},"last":10.15}]},{"osd":6,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":26.797999999999998,"5min":26.663,"15min":31.244},"min":{"1min":13.439,"5min":11.666,"15min":9.9979999999999993},"max":{"1min":100.47799999999999,"5min":229.86199999999999,"15min":229.86199999999999},"last":17.167999999999999},{"interface":"front","average":{"1min":31.841999999999999,"5min":24.629000000000001,"15min":27.375},"min":{"1min":8.6280000000000001,"5min":8.1159999999999997,"15min":7.4770000000000003},"max":{"1min":165.46000000000001,"5min":234.31100000000001,"15min":234.31100000000001},"last":15.792999999999999}]},{"osd":7,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":25.68,"5min":20.594999999999999,"15min":26.533999999999999},"min":{"1min":7.2679999999999998,"5min":6.2370000000000001,"15min":6.2370000000000001},"max":{"1min":67.870999999999995,"5min":195.94399999999999,"15min":195.94399999999999},"last":15.132},{"interface":"front","average":{"1min":23.308,"5min":25.666,"15min":26.741},"min":{"1min":11.151,"5min":11.151,"15min":7.3289999999999997},"max":{"1min":57.329000000000001,"5min":185.74700000000001,"15min":224.99600000000001},"last":11.151}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-25T05:26:17.224 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-25T05:26:17.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.335+0000 7fb9867b0640 10 monclient: get_monmap_and_config 2023-11-25T05:26:17.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.335+0000 7fb9867b0640 10 monclient: build_initial_monmap 2023-11-25T05:26:17.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.335+0000 7fb9867b0640 10 monclient: monmap: 2023-11-25T05:26:17.336 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:17.337 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:17.337 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:17.336199+0000 2023-11-25T05:26:17.337 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:17.336199+0000 2023-11-25T05:26:17.337 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:17.337 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:17.337 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:17.337 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:17.337 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:17.337 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:17.337 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:17.337 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:17.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 1 Processor -- start 2023-11-25T05:26:17.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 1 -- start start 2023-11-25T05:26:17.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 10 monclient: init 2023-11-25T05:26:17.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:17.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-11-25T05:26:17.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb9800a8910 0x7fb98013ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:17.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 10 monclient(hunting): picked mon.noname-a con 0x7fb9800a8910 addr v2:172.21.15.7:3300/0 2023-11-25T05:26:17.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb9800a9b70 addr v1:172.21.15.90:6789/0 2023-11-25T05:26:17.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb980141f90 0x7fb9801423f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:17.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 10 monclient(hunting): picked mon.noname-b con 0x7fb980141f90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:17.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9800a8cf0 con 0x7fb9800a9b70 2023-11-25T05:26:17.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:17.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9800aabd0 con 0x7fb9800a8910 2023-11-25T05:26:17.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:17.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800aaeb0 con 0x7fb980141f90 2023-11-25T05:26:17.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:17.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.336+0000 7fb9867b0640 10 monclient(hunting): authenticate will time out at 2703.579517s 2023-11-25T05:26:17.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.337+0000 7fb984d4d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb980141f90 0x7fb9801423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.337+0000 7fb984d4d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb980141f90 0x7fb9801423f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38174/0 (socket says 172.21.15.7:38174) 2023-11-25T05:26:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.337+0000 7fb984d4d640 1 -- 172.21.15.7:0/2042437972 learned_addr learned my addr 172.21.15.7:0/2042437972 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.337+0000 7fb984d4d640 10 monclient(hunting): get_auth_request con 0x7fb980141f90 auth_method 0 2023-11-25T05:26:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.337+0000 7fb984d4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.337+0000 7fb984d4d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.337+0000 7fb984d4d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.338+0000 7fb984d4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.338+0000 7fb984d4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.338+0000 7fb984d4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.339+0000 7fb97ffff640 1 --2- 172.21.15.7:0/2042437972 >> v2:172.21.15.7:3300/0 conn(0x7fb9800a8910 0x7fb98013ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.339+0000 7fb984d4d640 10 monclient(hunting): handle_auth_done global_id 17363 payload 274 2023-11-25T05:26:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.339+0000 7fb984d4d640 1 -- 172.21.15.7:0/2042437972 >> v2:172.21.15.7:3300/0 conn(0x7fb9800a8910 msgr2=0x7fb98013ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.339+0000 7fb984d4d640 1 --2- 172.21.15.7:0/2042437972 >> v2:172.21.15.7:3300/0 conn(0x7fb9800a8910 0x7fb98013ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.339+0000 7fb984d4d640 1 -- 172.21.15.7:0/2042437972 >> v1:172.21.15.90:6789/0 conn(0x7fb9800a9b70 legacy=0x7fb98013f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.339+0000 7fb984d4d640 10 monclient: _finish_hunting 0 2023-11-25T05:26:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.339+0000 7fb984d4d640 1 monclient: found mon.noname-b 2023-11-25T05:26:17.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.339+0000 7fb984d4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:17.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.339+0000 7fb984d4d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:26:17.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.339+0000 7fb984d4d640 1 -- 172.21.15.7:0/2042437972 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9800a9720 con 0x7fb980141f90 2023-11-25T05:26:17.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.341+0000 7fb984d4d640 1 --2- 172.21.15.7:0/2042437972 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb980141f90 0x7fb9801423f0 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7fb978009e50 tx=0x7fb97802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe7a0412ff5ebda8 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb97effd640 1 -- 172.21.15.7:0/2042437972 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97802cb00 con 0x7fb980141f90 2023-11-25T05:26:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb97effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb97effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb97effd640 10 monclient: dump: 2023-11-25T05:26:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:17.348 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb97effd640 10 monclient: _finish_auth 0 2023-11-25T05:26:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb97effd640 10 monclient: _check_auth_tickets 2023-11-25T05:26:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb97effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb97effd640 1 -- 172.21.15.7:0/2042437972 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97802cc60 con 0x7fb980141f90 2023-11-25T05:26:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb97effd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb97effd640 1 -- 172.21.15.7:0/2042437972 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9780343f0 con 0x7fb980141f90 2023-11-25T05:26:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb9867b0640 5 monclient: authenticate success, global_id 17363 2023-11-25T05:26:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb9867b0640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb97effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb9867b0640 4 set_mon_vals no callback set 2023-11-25T05:26:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb97effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:26:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.345+0000 7fb97effd640 10 monclient: dump: 2023-11-25T05:26:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:17.349 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:17.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.346+0000 7fb9867b0640 10 monclient: shutdown 2023-11-25T05:26:17.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.346+0000 7fb9867b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:17.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.346+0000 7fb9867b0640 1 -- 172.21.15.7:0/2042437972 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb980141f90 msgr2=0x7fb9801423f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.346+0000 7fb9867b0640 1 --2- 172.21.15.7:0/2042437972 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb980141f90 0x7fb9801423f0 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7fb978009e50 tx=0x7fb97802bf60 comp rx=0 tx=0).stop 2023-11-25T05:26:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.346+0000 7fb9867b0640 1 -- 172.21.15.7:0/2042437972 shutdown_connections 2023-11-25T05:26:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.346+0000 7fb9867b0640 1 --2- 172.21.15.7:0/2042437972 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb980141f90 0x7fb9801423f0 unknown :-1 s=CLOSED pgs=3061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.346+0000 7fb9867b0640 1 --2- 172.21.15.7:0/2042437972 >> v2:172.21.15.7:3300/0 conn(0x7fb9800a8910 0x7fb98013ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.346+0000 7fb9867b0640 1 -- 172.21.15.7:0/2042437972 >> 172.21.15.7:0/2042437972 conn(0x7fb9800b7780 msgr2=0x7fb9800b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.346+0000 7fb9867b0640 1 -- 172.21.15.7:0/2042437972 shutdown_connections 2023-11-25T05:26:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.346+0000 7fb9867b0640 1 -- 172.21.15.7:0/2042437972 wait complete. 2023-11-25T05:26:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.347+0000 7fb9867b0640 10 monclient: build_initial_monmap 2023-11-25T05:26:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.347+0000 7fb9867b0640 10 monclient: monmap: 2023-11-25T05:26:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:17.353 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:17.354 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.347+0000 7fb9867b0640 1 librados: starting msgr at 2023-11-25T05:26:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.347+0000 7fb9867b0640 1 librados: starting objecter 2023-11-25T05:26:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.347+0000 7fb9867b0640 1 Processor -- start 2023-11-25T05:26:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 1 -- start start 2023-11-25T05:26:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 1 librados: setting wanted keys 2023-11-25T05:26:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 1 librados: calling monclient init 2023-11-25T05:26:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 10 monclient: init 2023-11-25T05:26:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-25T05:26:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9800a8910 0x7fb9800ab610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 10 monclient(hunting): picked mon.noname-a con 0x7fb9800a8910 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb980141f90 0x7fb9801334a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 10 monclient(hunting): picked mon.noname-b con 0x7fb980141f90 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb9801339e0 0x7fb980133e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 10 monclient(hunting): picked mon.noname-c con 0x7fb9801339e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800ac2b0 con 0x7fb9800a8910 2023-11-25T05:26:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9801343b0 con 0x7fb980141f90 2023-11-25T05:26:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800ad2d0 con 0x7fb9801339e0 2023-11-25T05:26:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.348+0000 7fb9867b0640 10 monclient(hunting): authenticate will time out at 2703.591450s 2023-11-25T05:26:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb984d4d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb9801339e0 0x7fb980133e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb97f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb980141f90 0x7fb9801334a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb984d4d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb9801339e0 0x7fb980133e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38186/0 (socket says 172.21.15.7:38186) 2023-11-25T05:26:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb984d4d640 1 -- 172.21.15.7:0/2830288420 learned_addr learned my addr 172.21.15.7:0/2830288420 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb984d4d640 10 monclient(hunting): get_auth_request con 0x7fb9801339e0 auth_method 0 2023-11-25T05:26:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb984d4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb984d4d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb984d4d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb97f7fe640 10 monclient(hunting): get_auth_request con 0x7fb980141f90 auth_method 0 2023-11-25T05:26:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb97f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb97f7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb97f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:17.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb984d4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:17.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb984d4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:17.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb984d4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:17.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb97f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:17.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb97f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:17.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.349+0000 7fb97f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:17.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.351+0000 7fb97ffff640 1 --2- 172.21.15.7:0/2830288420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9800a8910 0x7fb9800ab610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:17.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.352+0000 7fb97ffff640 10 monclient(hunting): get_auth_request con 0x7fb9800a8910 auth_method 0 2023-11-25T05:26:17.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.352+0000 7fb97ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:17.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.352+0000 7fb97ffff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:17.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.352+0000 7fb97ffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:17.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.352+0000 7fb97f7fe640 10 monclient(hunting): handle_auth_done global_id 21214 payload 824 2023-11-25T05:26:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.352+0000 7fb97f7fe640 1 -- 172.21.15.7:0/2830288420 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb9801339e0 msgr2=0x7fb980133e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.352+0000 7fb97f7fe640 1 --2- 172.21.15.7:0/2830288420 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb9801339e0 0x7fb980133e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.352+0000 7fb97f7fe640 1 -- 172.21.15.7:0/2830288420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9800a8910 msgr2=0x7fb9800ab610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.352+0000 7fb97f7fe640 1 --2- 172.21.15.7:0/2830288420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9800a8910 0x7fb9800ab610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.352+0000 7fb97f7fe640 10 monclient: _finish_hunting 0 2023-11-25T05:26:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.352+0000 7fb97f7fe640 1 monclient: found mon.noname-b 2023-11-25T05:26:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.352+0000 7fb97f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.352+0000 7fb97f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:26:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.352+0000 7fb97f7fe640 1 -- 172.21.15.7:0/2830288420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb978009b30 con 0x7fb980141f90 2023-11-25T05:26:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.354+0000 7fb97f7fe640 1 --2- 172.21.15.7:0/2830288420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb980141f90 0x7fb9801334a0 secure :-1 s=READY pgs=5144 cs=0 l=1 rev1=1 crypto rx=0x7fb97400c370 tx=0x7fb97400c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.354+0000 7fb97cff9640 10 client.?.objecter ms_handle_connect 0x7fb980141f90 2023-11-25T05:26:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.354+0000 7fb97cff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb97cff9640 1 -- 172.21.15.7:0/2830288420 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9740052d0 con 0x7fb980141f90 2023-11-25T05:26:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb97cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb97cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb97cff9640 10 monclient: dump: 2023-11-25T05:26:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:17.364 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb97cff9640 10 monclient: _finish_auth 0 2023-11-25T05:26:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb97cff9640 10 monclient: _check_auth_tickets 2023-11-25T05:26:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb97cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb97cff9640 1 -- 172.21.15.7:0/2830288420 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb974005430 con 0x7fb980141f90 2023-11-25T05:26:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb97cff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb97cff9640 1 -- 172.21.15.7:0/2830288420 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb974003850 con 0x7fb980141f90 2023-11-25T05:26:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb9867b0640 5 monclient: authenticate success, global_id 21214 2023-11-25T05:26:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb9867b0640 10 monclient: _renew_subs 2023-11-25T05:26:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb9867b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb9867b0640 1 -- 172.21.15.7:0/2830288420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb98013a940 con 0x7fb980141f90 2023-11-25T05:26:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb97cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb97cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb97cff9640 10 monclient: dump: 2023-11-25T05:26:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:17.367 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb97e7fc640 4 set_mon_vals no callback set 2023-11-25T05:26:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb9867b0640 10 client.21214.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb9867b0640 10 monclient: _renew_subs 2023-11-25T05:26:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb9867b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb9867b0640 1 -- 172.21.15.7:0/2830288420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9800abed0 con 0x7fb980141f90 2023-11-25T05:26:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.358+0000 7fb9867b0640 1 librados: init done 2023-11-25T05:26:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.359+0000 7fb9867b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.359+0000 7fb9867b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:17.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.359+0000 7fb9867b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:17.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.359+0000 7fb9867b0640 1 -- 172.21.15.7:0/2830288420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb98013a390 con 0x7fb980141f90 2023-11-25T05:26:17.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.377+0000 7fb97cff9640 1 -- 172.21.15.7:0/2830288420 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb9740039b0 con 0x7fb980141f90 2023-11-25T05:26:17.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.377+0000 7fb97cff9640 1 --2- 172.21.15.7:0/2830288420 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb96c06c7d0 0x7fb96c06ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:17.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.377+0000 7fb97ffff640 1 --2- 172.21.15.7:0/2830288420 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb96c06c7d0 0x7fb96c06ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:17.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.377+0000 7fb97ffff640 10 monclient: get_auth_request con 0x7fb96c06c7d0 auth_method 0 2023-11-25T05:26:17.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.378+0000 7fb97ffff640 1 --2- 172.21.15.7:0/2830288420 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb96c06c7d0 0x7fb96c06ec90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb97000a970 tx=0x7fb970008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:17.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.378+0000 7fb97cff9640 10 client.21214.objecter ms_handle_connect 0x7fb96c06c7d0 2023-11-25T05:26:17.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.379+0000 7fb97cff9640 1 -- 172.21.15.7:0/2830288420 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 1..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7fb974057c80 con 0x7fb980141f90 2023-11-25T05:26:17.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.379+0000 7fb97cff9640 10 client.21214.objecter ms_dispatch 0x7fb980000b90 osd_map(918..918 src has 1..918) v4 2023-11-25T05:26:17.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.379+0000 7fb97cff9640 3 client.21214.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:17.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.379+0000 7fb97cff9640 3 client.21214.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:17.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.379+0000 7fb97cff9640 20 client.21214.objecter dump_active .. 0 homeless 2023-11-25T05:26:17.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.618+0000 7fb97cff9640 1 -- 172.21.15.7:0/2830288420 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb974053fc0 con 0x7fb980141f90 2023-11-25T05:26:17.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.618+0000 7fb97cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:17.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.618+0000 7fb97cff9640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:26:17.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.800+0000 7fb9867b0640 1 -- 172.21.15.7:0/2830288420 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb9800b7310 con 0x7fb96c06c7d0 2023-11-25T05:26:17.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.805+0000 7fb97cff9640 1 -- 172.21.15.7:0/2830288420 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152053 (secure 0 0 0) 0x7fb9800b7310 con 0x7fb96c06c7d0 2023-11-25T05:26:17.808 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:26:17.808 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-25T05:26:17.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.807+0000 7fb9867b0640 10 librados: watch_flush enter 2023-11-25T05:26:17.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.807+0000 7fb9867b0640 10 librados: watch_flush exit 2023-11-25T05:26:17.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.807+0000 7fb9867b0640 20 client.21214.objecter shutdown clearing up homeless session... 2023-11-25T05:26:17.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.807+0000 7fb9867b0640 10 client.21214.objecter successfully canceled tick 2023-11-25T05:26:17.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.807+0000 7fb9867b0640 1 -- 172.21.15.7:0/2830288420 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb96c06c7d0 msgr2=0x7fb96c06ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:17.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.807+0000 7fb9867b0640 1 --2- 172.21.15.7:0/2830288420 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb96c06c7d0 0x7fb96c06ec90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb97000a970 tx=0x7fb970008040 comp rx=0 tx=0).stop 2023-11-25T05:26:17.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.807+0000 7fb9867b0640 10 monclient: shutdown 2023-11-25T05:26:17.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.807+0000 7fb9867b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:17.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.807+0000 7fb9867b0640 1 -- 172.21.15.7:0/2830288420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb980141f90 msgr2=0x7fb9801334a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:17.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.807+0000 7fb9867b0640 1 --2- 172.21.15.7:0/2830288420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb980141f90 0x7fb9801334a0 secure :-1 s=READY pgs=5144 cs=0 l=1 rev1=1 crypto rx=0x7fb97400c370 tx=0x7fb97400c830 comp rx=0 tx=0).stop 2023-11-25T05:26:17.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.808+0000 7fb9867b0640 1 -- 172.21.15.7:0/2830288420 shutdown_connections 2023-11-25T05:26:17.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.808+0000 7fb9867b0640 1 --2- 172.21.15.7:0/2830288420 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb96c06c7d0 0x7fb96c06ec90 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.808+0000 7fb9867b0640 1 --2- 172.21.15.7:0/2830288420 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb9801339e0 0x7fb980133e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.808+0000 7fb9867b0640 1 --2- 172.21.15.7:0/2830288420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb980141f90 0x7fb9801334a0 unknown :-1 s=CLOSED pgs=5144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.808+0000 7fb9867b0640 1 --2- 172.21.15.7:0/2830288420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9800a8910 0x7fb9800ab610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.808+0000 7fb9867b0640 1 -- 172.21.15.7:0/2830288420 >> 172.21.15.7:0/2830288420 conn(0x7fb9800b7780 msgr2=0x7fb98013f180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:17.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.808+0000 7fb9867b0640 1 -- 172.21.15.7:0/2830288420 shutdown_connections 2023-11-25T05:26:17.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.808+0000 7fb9867b0640 1 -- 172.21.15.7:0/2830288420 wait complete. 2023-11-25T05:26:17.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.808+0000 7fb9867b0640 1 librados: shutdown 2023-11-25T05:26:17.820 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1580,"stamp":"2023-11-25T05:26:17.713644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":139,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":596648,"kb_used_data":106944,"kb_used_omap":12,"kb_used_meta":489459,"kb_avail":754378072,"statfs":{"total":773094113280,"available":772483145728,"internally_reserved":0,"allocated":109510656,"data_stored":67065247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":501206579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":389,"apply_latency_ms":389,"commit_latency_ns":389000000,"apply_latency_ns":389000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.158866"},"pg_stats":[{"pgid":"9.15","version":"0'0","reported_seq":861,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.913594+0000","last_change":"2023-11-25T05:05:46.884752+0000","last_active":"2023-11-25T05:26:15.913594+0000","last_peered":"2023-11-25T05:26:15.913594+0000","last_clean":"2023-11-25T05:26:15.913594+0000","last_became_active":"2023-11-25T05:05:46.875394+0000","last_became_peered":"2023-11-25T05:05:46.875394+0000","last_unstale":"2023-11-25T05:26:15.913594+0000","last_undegraded":"2023-11-25T05:26:15.913594+0000","last_fullsized":"2023-11-25T05:26:15.913594+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:08:59.865717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":870,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.130927+0000","last_change":"2023-11-25T05:05:46.929874+0000","last_active":"2023-11-25T05:26:16.130927+0000","last_peered":"2023-11-25T05:26:16.130927+0000","last_clean":"2023-11-25T05:26:16.130927+0000","last_became_active":"2023-11-25T05:05:46.923663+0000","last_became_peered":"2023-11-25T05:05:46.923663+0000","last_unstale":"2023-11-25T05:26:16.130927+0000","last_undegraded":"2023-11-25T05:26:16.130927+0000","last_fullsized":"2023-11-25T05:26:16.130927+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:38:50.120101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.706495+0000","last_change":"2023-11-25T05:05:51.087897+0000","last_active":"2023-11-25T05:26:12.706495+0000","last_peered":"2023-11-25T05:26:12.706495+0000","last_clean":"2023-11-25T05:26:12.706495+0000","last_became_active":"2023-11-25T05:05:50.968719+0000","last_became_peered":"2023-11-25T05:05:50.968719+0000","last_unstale":"2023-11-25T05:26:12.706495+0000","last_undegraded":"2023-11-25T05:26:12.706495+0000","last_fullsized":"2023-11-25T05:26:12.706495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:28:52.119629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":850,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.177301+0000","last_change":"2023-11-25T05:06:05.094337+0000","last_active":"2023-11-25T05:26:14.177301+0000","last_peered":"2023-11-25T05:26:14.177301+0000","last_clean":"2023-11-25T05:26:14.177301+0000","last_became_active":"2023-11-25T05:06:05.084111+0000","last_became_peered":"2023-11-25T05:06:05.084111+0000","last_unstale":"2023-11-25T05:26:14.177301+0000","last_undegraded":"2023-11-25T05:26:14.177301+0000","last_fullsized":"2023-11-25T05:26:14.177301+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T13:39:56.459513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":861,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.898941+0000","last_change":"2023-11-25T05:05:47.892163+0000","last_active":"2023-11-25T05:26:15.898941+0000","last_peered":"2023-11-25T05:26:15.898941+0000","last_clean":"2023-11-25T05:26:15.898941+0000","last_became_active":"2023-11-25T05:05:47.855679+0000","last_became_peered":"2023-11-25T05:05:47.855679+0000","last_unstale":"2023-11-25T05:26:15.898941+0000","last_undegraded":"2023-11-25T05:26:15.898941+0000","last_fullsized":"2023-11-25T05:26:15.898941+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:54:11.446291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":874,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.189467+0000","last_change":"2023-11-25T05:06:09.646267+0000","last_active":"2023-11-25T05:26:14.189467+0000","last_peered":"2023-11-25T05:26:14.189467+0000","last_clean":"2023-11-25T05:26:14.189467+0000","last_became_active":"2023-11-25T05:06:09.633454+0000","last_became_peered":"2023-11-25T05:06:09.633454+0000","last_unstale":"2023-11-25T05:26:14.189467+0000","last_undegraded":"2023-11-25T05:26:14.189467+0000","last_fullsized":"2023-11-25T05:26:14.189467+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:36:34.447428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":863,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.589752+0000","last_change":"2023-11-25T05:05:53.105351+0000","last_active":"2023-11-25T05:26:12.589752+0000","last_peered":"2023-11-25T05:26:12.589752+0000","last_clean":"2023-11-25T05:26:12.589752+0000","last_became_active":"2023-11-25T05:05:53.099253+0000","last_became_peered":"2023-11-25T05:05:53.099253+0000","last_unstale":"2023-11-25T05:26:12.589752+0000","last_undegraded":"2023-11-25T05:26:12.589752+0000","last_fullsized":"2023-11-25T05:26:12.589752+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:25:34.409386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":874,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.210137+0000","last_change":"2023-11-25T05:05:43.502596+0000","last_active":"2023-11-25T05:26:14.210137+0000","last_peered":"2023-11-25T05:26:14.210137+0000","last_clean":"2023-11-25T05:26:14.210137+0000","last_became_active":"2023-11-25T05:05:43.496305+0000","last_became_peered":"2023-11-25T05:05:43.496305+0000","last_unstale":"2023-11-25T05:26:14.210137+0000","last_undegraded":"2023-11-25T05:26:14.210137+0000","last_fullsized":"2023-11-25T05:26:14.210137+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:56:11.421935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":869,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.390854+0000","last_change":"2023-11-25T05:05:42.708461+0000","last_active":"2023-11-25T05:26:11.390854+0000","last_peered":"2023-11-25T05:26:11.390854+0000","last_clean":"2023-11-25T05:26:11.390854+0000","last_became_active":"2023-11-25T05:05:42.672388+0000","last_became_peered":"2023-11-25T05:05:42.672388+0000","last_unstale":"2023-11-25T05:26:11.390854+0000","last_undegraded":"2023-11-25T05:26:11.390854+0000","last_fullsized":"2023-11-25T05:26:11.390854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:34:44.252922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":870,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.132522+0000","last_change":"2023-11-25T05:05:46.966035+0000","last_active":"2023-11-25T05:26:16.132522+0000","last_peered":"2023-11-25T05:26:16.132522+0000","last_clean":"2023-11-25T05:26:16.132522+0000","last_became_active":"2023-11-25T05:05:46.959854+0000","last_became_peered":"2023-11-25T05:05:46.959854+0000","last_unstale":"2023-11-25T05:26:16.132522+0000","last_undegraded":"2023-11-25T05:26:16.132522+0000","last_fullsized":"2023-11-25T05:26:16.132522+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:00:30.622747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":869,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.400561+0000","last_change":"2023-11-25T05:05:47.759205+0000","last_active":"2023-11-25T05:26:11.400561+0000","last_peered":"2023-11-25T05:26:11.400561+0000","last_clean":"2023-11-25T05:26:11.400561+0000","last_became_active":"2023-11-25T05:05:47.753176+0000","last_became_peered":"2023-11-25T05:05:47.753176+0000","last_unstale":"2023-11-25T05:26:11.400561+0000","last_undegraded":"2023-11-25T05:26:11.400561+0000","last_fullsized":"2023-11-25T05:26:11.400561+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:04:21.487913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":870,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.427728+0000","last_change":"2023-11-25T05:05:45.604133+0000","last_active":"2023-11-25T05:26:11.427728+0000","last_peered":"2023-11-25T05:26:11.427728+0000","last_clean":"2023-11-25T05:26:11.427728+0000","last_became_active":"2023-11-25T05:05:45.597733+0000","last_became_peered":"2023-11-25T05:05:45.597733+0000","last_unstale":"2023-11-25T05:26:11.427728+0000","last_undegraded":"2023-11-25T05:26:11.427728+0000","last_fullsized":"2023-11-25T05:26:11.427728+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:20:05.883993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.604334+0000","last_change":"2023-11-25T05:05:47.799603+0000","last_active":"2023-11-25T05:26:13.604334+0000","last_peered":"2023-11-25T05:26:13.604334+0000","last_clean":"2023-11-25T05:26:13.604334+0000","last_became_active":"2023-11-25T05:05:47.792727+0000","last_became_peered":"2023-11-25T05:05:47.792727+0000","last_unstale":"2023-11-25T05:26:13.604334+0000","last_undegraded":"2023-11-25T05:26:13.604334+0000","last_fullsized":"2023-11-25T05:26:13.604334+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:50:46.409556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":866,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.923391+0000","last_change":"2023-11-25T05:04:33.582978+0000","last_active":"2023-11-25T05:26:15.923391+0000","last_peered":"2023-11-25T05:26:15.923391+0000","last_clean":"2023-11-25T05:26:15.923391+0000","last_became_active":"2023-11-25T05:04:33.577510+0000","last_became_peered":"2023-11-25T05:04:33.577510+0000","last_unstale":"2023-11-25T05:26:15.923391+0000","last_undegraded":"2023-11-25T05:26:15.923391+0000","last_fullsized":"2023-11-25T05:26:15.923391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:53:20.091826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":862,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.911955+0000","last_change":"2023-11-25T05:05:58.700776+0000","last_active":"2023-11-25T05:26:15.911955+0000","last_peered":"2023-11-25T05:26:15.911955+0000","last_clean":"2023-11-25T05:26:15.911955+0000","last_became_active":"2023-11-25T05:05:58.695110+0000","last_became_peered":"2023-11-25T05:05:58.695110+0000","last_unstale":"2023-11-25T05:26:15.911955+0000","last_undegraded":"2023-11-25T05:26:15.911955+0000","last_fullsized":"2023-11-25T05:26:15.911955+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T05:50:22.654582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":875,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.715982+0000","last_change":"2023-11-25T05:04:33.645680+0000","last_active":"2023-11-25T05:26:13.715982+0000","last_peered":"2023-11-25T05:26:13.715982+0000","last_clean":"2023-11-25T05:26:13.715982+0000","last_became_active":"2023-11-25T05:04:33.610776+0000","last_became_peered":"2023-11-25T05:04:33.610776+0000","last_unstale":"2023-11-25T05:26:13.715982+0000","last_undegraded":"2023-11-25T05:26:13.715982+0000","last_fullsized":"2023-11-25T05:26:13.715982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:02:08.013900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.680167+0000","last_change":"2023-11-25T05:06:12.667457+0000","last_active":"2023-11-25T05:26:12.680167+0000","last_peered":"2023-11-25T05:26:12.680167+0000","last_clean":"2023-11-25T05:26:12.680167+0000","last_became_active":"2023-11-25T05:06:12.543810+0000","last_became_peered":"2023-11-25T05:06:12.543810+0000","last_unstale":"2023-11-25T05:26:12.680167+0000","last_undegraded":"2023-11-25T05:26:12.680167+0000","last_fullsized":"2023-11-25T05:26:12.680167+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:13:23.439209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.910285+0000","last_change":"2023-11-25T05:04:33.350255+0000","last_active":"2023-11-25T05:26:15.910285+0000","last_peered":"2023-11-25T05:26:15.910285+0000","last_clean":"2023-11-25T05:26:15.910285+0000","last_became_active":"2023-11-25T05:04:33.337158+0000","last_became_peered":"2023-11-25T05:04:33.337158+0000","last_unstale":"2023-11-25T05:26:15.910285+0000","last_undegraded":"2023-11-25T05:26:15.910285+0000","last_fullsized":"2023-11-25T05:26:15.910285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:25:34.365788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.708724+0000","last_change":"2023-11-25T05:05:51.120038+0000","last_active":"2023-11-25T05:26:12.708724+0000","last_peered":"2023-11-25T05:26:12.708724+0000","last_clean":"2023-11-25T05:26:12.708724+0000","last_became_active":"2023-11-25T05:05:51.097150+0000","last_became_peered":"2023-11-25T05:05:51.097150+0000","last_unstale":"2023-11-25T05:26:12.708724+0000","last_undegraded":"2023-11-25T05:26:12.708724+0000","last_fullsized":"2023-11-25T05:26:12.708724+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:34:02.990987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":876,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.134189+0000","last_change":"2023-11-25T05:04:33.396056+0000","last_active":"2023-11-25T05:26:16.134189+0000","last_peered":"2023-11-25T05:26:16.134189+0000","last_clean":"2023-11-25T05:26:16.134189+0000","last_became_active":"2023-11-25T05:04:33.388849+0000","last_became_peered":"2023-11-25T05:04:33.388849+0000","last_unstale":"2023-11-25T05:26:16.134189+0000","last_undegraded":"2023-11-25T05:26:16.134189+0000","last_fullsized":"2023-11-25T05:26:16.134189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:11:08.763309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":871,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.120125+0000","last_change":"2023-11-25T05:05:55.046489+0000","last_active":"2023-11-25T05:26:16.120125+0000","last_peered":"2023-11-25T05:26:16.120125+0000","last_clean":"2023-11-25T05:26:16.120125+0000","last_became_active":"2023-11-25T05:05:55.038711+0000","last_became_peered":"2023-11-25T05:05:55.038711+0000","last_unstale":"2023-11-25T05:26:16.120125+0000","last_undegraded":"2023-11-25T05:26:16.120125+0000","last_fullsized":"2023-11-25T05:26:16.120125+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:43:58.709876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.902474+0000","last_change":"2023-11-25T05:04:33.381810+0000","last_active":"2023-11-25T05:26:15.902474+0000","last_peered":"2023-11-25T05:26:15.902474+0000","last_clean":"2023-11-25T05:26:15.902474+0000","last_became_active":"2023-11-25T05:04:33.376375+0000","last_became_peered":"2023-11-25T05:04:33.376375+0000","last_unstale":"2023-11-25T05:26:15.902474+0000","last_undegraded":"2023-11-25T05:26:15.902474+0000","last_fullsized":"2023-11-25T05:26:15.902474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:41:46.038487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":875,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.207787+0000","last_change":"2023-11-25T05:05:44.569896+0000","last_active":"2023-11-25T05:26:14.207787+0000","last_peered":"2023-11-25T05:26:14.207787+0000","last_clean":"2023-11-25T05:26:14.207787+0000","last_became_active":"2023-11-25T05:05:44.554983+0000","last_became_peered":"2023-11-25T05:05:44.554983+0000","last_unstale":"2023-11-25T05:26:14.207787+0000","last_undegraded":"2023-11-25T05:26:14.207787+0000","last_fullsized":"2023-11-25T05:26:14.207787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:59:24.121163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":871,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.564292+0000","last_change":"2023-11-25T05:04:33.238092+0000","last_active":"2023-11-25T05:26:13.564292+0000","last_peered":"2023-11-25T05:26:13.564292+0000","last_clean":"2023-11-25T05:26:13.564292+0000","last_became_active":"2023-11-25T05:04:33.196533+0000","last_became_peered":"2023-11-25T05:04:33.196533+0000","last_unstale":"2023-11-25T05:26:13.564292+0000","last_undegraded":"2023-11-25T05:26:13.564292+0000","last_fullsized":"2023-11-25T05:26:13.564292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:06:22.475692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":871,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.126082+0000","last_change":"2023-11-25T05:05:45.774913+0000","last_active":"2023-11-25T05:26:16.126082+0000","last_peered":"2023-11-25T05:26:16.126082+0000","last_clean":"2023-11-25T05:26:16.126082+0000","last_became_active":"2023-11-25T05:05:45.759824+0000","last_became_peered":"2023-11-25T05:05:45.759824+0000","last_unstale":"2023-11-25T05:26:16.126082+0000","last_undegraded":"2023-11-25T05:26:16.126082+0000","last_fullsized":"2023-11-25T05:26:16.126082+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:41:12.473852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.887432+0000","last_change":"2023-11-25T05:04:33.294027+0000","last_active":"2023-11-25T05:26:15.887432+0000","last_peered":"2023-11-25T05:26:15.887432+0000","last_clean":"2023-11-25T05:26:15.887432+0000","last_became_active":"2023-11-25T05:04:33.242068+0000","last_became_peered":"2023-11-25T05:04:33.242068+0000","last_unstale":"2023-11-25T05:26:15.887432+0000","last_undegraded":"2023-11-25T05:26:15.887432+0000","last_fullsized":"2023-11-25T05:26:15.887432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:37:54.756624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":871,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.106772+0000","last_change":"2023-11-25T05:05:45.087189+0000","last_active":"2023-11-25T05:26:16.106772+0000","last_peered":"2023-11-25T05:26:16.106772+0000","last_clean":"2023-11-25T05:26:16.106772+0000","last_became_active":"2023-11-25T05:05:45.057546+0000","last_became_peered":"2023-11-25T05:05:45.057546+0000","last_unstale":"2023-11-25T05:26:16.106772+0000","last_undegraded":"2023-11-25T05:26:16.106772+0000","last_fullsized":"2023-11-25T05:26:16.106772+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:02:04.146157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.907191+0000","last_change":"2023-11-25T05:04:33.784328+0000","last_active":"2023-11-25T05:26:15.907191+0000","last_peered":"2023-11-25T05:26:15.907191+0000","last_clean":"2023-11-25T05:26:15.907191+0000","last_became_active":"2023-11-25T05:04:33.778042+0000","last_became_peered":"2023-11-25T05:04:33.778042+0000","last_unstale":"2023-11-25T05:26:15.907191+0000","last_undegraded":"2023-11-25T05:26:15.907191+0000","last_fullsized":"2023-11-25T05:26:15.907191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:40:38.193703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":974,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.192116+0000","last_change":"2023-11-25T05:04:28.241367+0000","last_active":"2023-11-25T05:26:14.192116+0000","last_peered":"2023-11-25T05:26:14.192116+0000","last_clean":"2023-11-25T05:26:14.192116+0000","last_became_active":"2023-11-25T05:04:27.388156+0000","last_became_peered":"2023-11-25T05:04:27.388156+0000","last_unstale":"2023-11-25T05:26:14.192116+0000","last_undegraded":"2023-11-25T05:26:14.192116+0000","last_fullsized":"2023-11-25T05:26:14.192116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:22.310975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:22.310975+0000","last_clean_scrub_stamp":"2023-11-25T05:04:22.310975+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:50:37.122665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":870,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.404755+0000","last_change":"2023-11-25T05:05:47.996637+0000","last_active":"2023-11-25T05:26:11.404755+0000","last_peered":"2023-11-25T05:26:11.404755+0000","last_clean":"2023-11-25T05:26:11.404755+0000","last_became_active":"2023-11-25T05:05:47.973359+0000","last_became_peered":"2023-11-25T05:05:47.973359+0000","last_unstale":"2023-11-25T05:26:11.404755+0000","last_undegraded":"2023-11-25T05:26:11.404755+0000","last_fullsized":"2023-11-25T05:26:11.404755+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:05:25.523743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":852,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.171648+0000","last_change":"2023-11-25T05:05:54.064947+0000","last_active":"2023-11-25T05:26:14.171648+0000","last_peered":"2023-11-25T05:26:14.171648+0000","last_clean":"2023-11-25T05:26:14.171648+0000","last_became_active":"2023-11-25T05:05:54.045707+0000","last_became_peered":"2023-11-25T05:05:54.045707+0000","last_unstale":"2023-11-25T05:26:14.171648+0000","last_undegraded":"2023-11-25T05:26:14.171648+0000","last_fullsized":"2023-11-25T05:26:14.171648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T17:02:26.650184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":872,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.104094+0000","last_change":"2023-11-25T05:05:42.506167+0000","last_active":"2023-11-25T05:26:16.104094+0000","last_peered":"2023-11-25T05:26:16.104094+0000","last_clean":"2023-11-25T05:26:16.104094+0000","last_became_active":"2023-11-25T05:05:42.487757+0000","last_became_peered":"2023-11-25T05:05:42.487757+0000","last_unstale":"2023-11-25T05:26:16.104094+0000","last_undegraded":"2023-11-25T05:26:16.104094+0000","last_fullsized":"2023-11-25T05:26:16.104094+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:58:18.951826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"38'7","reported_seq":879,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.562585+0000","last_change":"2023-11-25T05:05:45.233028+0000","last_active":"2023-11-25T05:26:13.562585+0000","last_peered":"2023-11-25T05:26:13.562585+0000","last_clean":"2023-11-25T05:26:13.562585+0000","last_became_active":"2023-11-25T05:05:45.212870+0000","last_became_peered":"2023-11-25T05:05:45.212870+0000","last_unstale":"2023-11-25T05:26:13.562585+0000","last_undegraded":"2023-11-25T05:26:13.562585+0000","last_fullsized":"2023-11-25T05:26:13.562585+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:44:02.468772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.662501+0000","last_change":"2023-11-25T05:05:49.947608+0000","last_active":"2023-11-25T05:26:12.662501+0000","last_peered":"2023-11-25T05:26:12.662501+0000","last_clean":"2023-11-25T05:26:12.662501+0000","last_became_active":"2023-11-25T05:05:49.941593+0000","last_became_peered":"2023-11-25T05:05:49.941593+0000","last_unstale":"2023-11-25T05:26:12.662501+0000","last_undegraded":"2023-11-25T05:26:12.662501+0000","last_fullsized":"2023-11-25T05:26:12.662501+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:02:09.269520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.710445+0000","last_change":"2023-11-25T05:06:26.505899+0000","last_active":"2023-11-25T05:26:12.710445+0000","last_peered":"2023-11-25T05:26:12.710445+0000","last_clean":"2023-11-25T05:26:12.710445+0000","last_became_active":"2023-11-25T05:06:26.500023+0000","last_became_peered":"2023-11-25T05:06:26.500023+0000","last_unstale":"2023-11-25T05:26:12.710445+0000","last_undegraded":"2023-11-25T05:26:12.710445+0000","last_fullsized":"2023-11-25T05:26:12.710445+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T05:42:16.923983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.602721+0000","last_change":"2023-11-25T05:05:48.099018+0000","last_active":"2023-11-25T05:26:13.602721+0000","last_peered":"2023-11-25T05:26:13.602721+0000","last_clean":"2023-11-25T05:26:13.602721+0000","last_became_active":"2023-11-25T05:05:48.093083+0000","last_became_peered":"2023-11-25T05:05:48.093083+0000","last_unstale":"2023-11-25T05:26:13.602721+0000","last_undegraded":"2023-11-25T05:26:13.602721+0000","last_fullsized":"2023-11-25T05:26:13.602721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:36:31.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":870,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.139235+0000","last_change":"2023-11-25T05:05:55.013827+0000","last_active":"2023-11-25T05:26:16.139235+0000","last_peered":"2023-11-25T05:26:16.139235+0000","last_clean":"2023-11-25T05:26:16.139235+0000","last_became_active":"2023-11-25T05:05:55.006070+0000","last_became_peered":"2023-11-25T05:05:55.006070+0000","last_unstale":"2023-11-25T05:26:16.139235+0000","last_undegraded":"2023-11-25T05:26:16.139235+0000","last_fullsized":"2023-11-25T05:26:16.139235+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:42:44.573947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":870,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.123587+0000","last_change":"2023-11-25T05:05:42.334720+0000","last_active":"2023-11-25T05:26:16.123587+0000","last_peered":"2023-11-25T05:26:16.123587+0000","last_clean":"2023-11-25T05:26:16.123587+0000","last_became_active":"2023-11-25T05:05:42.327418+0000","last_became_peered":"2023-11-25T05:05:42.327418+0000","last_unstale":"2023-11-25T05:26:16.123587+0000","last_undegraded":"2023-11-25T05:26:16.123587+0000","last_fullsized":"2023-11-25T05:26:16.123587+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:45:59.399822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":869,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.679937+0000","last_change":"2023-11-25T05:05:45.341062+0000","last_active":"2023-11-25T05:26:13.679937+0000","last_peered":"2023-11-25T05:26:13.679937+0000","last_clean":"2023-11-25T05:26:13.679937+0000","last_became_active":"2023-11-25T05:05:45.334024+0000","last_became_peered":"2023-11-25T05:05:45.334024+0000","last_unstale":"2023-11-25T05:26:13.679937+0000","last_undegraded":"2023-11-25T05:26:13.679937+0000","last_fullsized":"2023-11-25T05:26:13.679937+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T05:06:56.822225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.582134+0000","last_change":"2023-11-25T05:06:27.132506+0000","last_active":"2023-11-25T05:26:12.582134+0000","last_peered":"2023-11-25T05:26:12.582134+0000","last_clean":"2023-11-25T05:26:12.582134+0000","last_became_active":"2023-11-25T05:06:27.122861+0000","last_became_peered":"2023-11-25T05:06:27.122861+0000","last_unstale":"2023-11-25T05:26:12.582134+0000","last_undegraded":"2023-11-25T05:26:12.582134+0000","last_fullsized":"2023-11-25T05:26:12.582134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:07:31.032076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":869,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.369980+0000","last_change":"2023-11-25T05:05:57.191530+0000","last_active":"2023-11-25T05:26:11.369980+0000","last_peered":"2023-11-25T05:26:11.369980+0000","last_clean":"2023-11-25T05:26:11.369980+0000","last_became_active":"2023-11-25T05:05:57.177615+0000","last_became_peered":"2023-11-25T05:05:57.177615+0000","last_unstale":"2023-11-25T05:26:11.369980+0000","last_undegraded":"2023-11-25T05:26:11.369980+0000","last_fullsized":"2023-11-25T05:26:11.369980+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:21:49.038926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":16,"seq":68719476999,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66640,"kb_used_data":13244,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305200,"statfs":{"total":96636764160,"available":96568524800,"internally_reserved":0,"allocated":13561856,"data_stored":8262098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":60,"apply_latency_ms":60,"commit_latency_ns":60000000,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":17.164000000000001,"5min":25.925000000000001,"15min":25.350999999999999},"min":{"1min":4.9020000000000001,"5min":4.9020000000000001,"15min":4.9020000000000001},"max":{"1min":100.782,"5min":164.57300000000001,"15min":203.28800000000001},"last":10.269},{"interface":"front","average":{"1min":15.237,"5min":26.946000000000002,"15min":26.018000000000001},"min":{"1min":5.7000000000000002,"5min":5.6859999999999999,"15min":5.4400000000000004},"max":{"1min":92.989000000000004,"5min":238.51300000000001,"15min":347.452},"last":17.202000000000002}]},{"osd":1,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":20.888000000000002,"5min":26.521999999999998,"15min":25.02},"min":{"1min":8.3249999999999993,"5min":7.4660000000000002,"15min":5.6589999999999998},"max":{"1min":86.962999999999994,"5min":287.27199999999999,"15min":287.27199999999999},"last":19.829999999999998},{"interface":"front","average":{"1min":17.350999999999999,"5min":27.222000000000001,"15min":23.896000000000001},"min":{"1min":5.282,"5min":5.282,"15min":5.282},"max":{"1min":57.453000000000003,"5min":286.57400000000001,"15min":286.57400000000001},"last":11.914}]},{"osd":3,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":23.404,"5min":20.637,"15min":20.605},"min":{"1min":6.3289999999999997,"5min":6.3289999999999997,"15min":5.9459999999999997},"max":{"1min":167.83799999999999,"5min":167.83799999999999,"15min":177.66},"last":9.5030000000000001},{"interface":"front","average":{"1min":30.977,"5min":28.187999999999999,"15min":31.155999999999999},"min":{"1min":11.647,"5min":8.1829999999999998,"15min":7.609},"max":{"1min":156.17699999999999,"5min":156.17699999999999,"15min":273.99900000000002},"last":15.653}]},{"osd":4,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":20.108000000000001,"5min":29.747,"15min":26.818000000000001},"min":{"1min":7.7789999999999999,"5min":6.9109999999999996,"15min":6.9109999999999996},"max":{"1min":71.611000000000004,"5min":275.66300000000001,"15min":349.90300000000002},"last":12.734999999999999},{"interface":"front","average":{"1min":17.427,"5min":26.164999999999999,"15min":22.431000000000001},"min":{"1min":7.1589999999999998,"5min":7.1589999999999998,"15min":6.6760000000000002},"max":{"1min":73.219999999999999,"5min":285.12700000000001,"15min":285.12700000000001},"last":7.8719999999999999}]},{"osd":5,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":22.722999999999999,"5min":24.5,"15min":29.382999999999999},"min":{"1min":11.710000000000001,"5min":9.7840000000000007,"15min":8.0619999999999994},"max":{"1min":89.492000000000004,"5min":138.34899999999999,"15min":353.63400000000001},"last":15.005000000000001},{"interface":"front","average":{"1min":26.366,"5min":30.738,"15min":34.677999999999997},"min":{"1min":14.381,"5min":13.541,"15min":11.539999999999999},"max":{"1min":96.620999999999995,"5min":146.41300000000001,"15min":354.56200000000001},"last":21.600999999999999}]},{"osd":6,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":31.605,"5min":30.536999999999999,"15min":31.172000000000001},"min":{"1min":16.321000000000002,"5min":11.292999999999999,"15min":9.7590000000000003},"max":{"1min":153.01400000000001,"5min":153.01400000000001,"15min":213.11000000000001},"last":20.943000000000001},{"interface":"front","average":{"1min":33.220999999999997,"5min":34.654000000000003,"15min":34.892000000000003},"min":{"1min":15.217000000000001,"5min":12.462999999999999,"15min":12.462999999999999},"max":{"1min":144.51400000000001,"5min":288.30000000000001,"15min":288.30000000000001},"last":22.247}]},{"osd":7,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":25.690999999999999,"5min":27.300000000000001,"15min":26.981000000000002},"min":{"1min":11.557,"5min":6.375,"15min":6.375},"max":{"1min":90.355999999999995,"5min":216.095,"15min":350.81599999999997},"last":17.853999999999999},{"interface":"front","average":{"1min":21.873000000000001,"5min":22.734000000000002,"15min":23.170000000000002},"min":{"1min":8.6400000000000006,"5min":7.5419999999999998,"15min":7.1669999999999998},"max":{"1min":96.100999999999999,"5min":141.59100000000001,"15min":210.55500000000001},"last":13.403}]}]},{"osd":1,"up_from":16,"seq":68719476999,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67056,"kb_used_data":13648,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94304784,"statfs":{"total":96636764160,"available":96568098816,"internally_reserved":0,"allocated":13975552,"data_stored":8666539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":23.388000000000002,"5min":20.16,"15min":21.632000000000001},"min":{"1min":6.165,"5min":5.9809999999999999,"15min":5.1600000000000001},"max":{"1min":163.62,"5min":221.589,"15min":349.75799999999998},"last":35.331000000000003},{"interface":"front","average":{"1min":32.435000000000002,"5min":29.16,"15min":26.466999999999999},"min":{"1min":9.4299999999999997,"5min":7.1529999999999996,"15min":6.5309999999999997},"max":{"1min":240.31200000000001,"5min":318.02600000000001,"15min":318.02600000000001},"last":28.692}]},{"osd":2,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":28.914000000000001,"5min":20.263000000000002,"15min":22.899999999999999},"min":{"1min":6.7889999999999997,"5min":5.1429999999999998,"15min":4.9619999999999997},"max":{"1min":148.28299999999999,"5min":161.10900000000001,"15min":353.286},"last":17.966000000000001},{"interface":"front","average":{"1min":34.442,"5min":21.100999999999999,"15min":19.414999999999999},"min":{"1min":7.7750000000000004,"5min":6.3280000000000003,"15min":5.7149999999999999},"max":{"1min":169.84100000000001,"5min":169.84100000000001,"15min":247.99199999999999},"last":30.224}]},{"osd":3,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":30.649000000000001,"5min":28.231999999999999,"15min":24.68},"min":{"1min":7.0339999999999998,"5min":6.665,"15min":6.5899999999999999},"max":{"1min":169.172,"5min":244.58199999999999,"15min":440.05599999999998},"last":33.240000000000002},{"interface":"front","average":{"1min":31.716000000000001,"5min":30.024000000000001,"15min":28.202999999999999},"min":{"1min":8.4369999999999994,"5min":8.4369999999999994,"15min":5.9889999999999999},"max":{"1min":168.58000000000001,"5min":242.29400000000001,"15min":383.39800000000002},"last":28.02}]},{"osd":4,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":20.422000000000001,"5min":17.603000000000002,"15min":21.190999999999999},"min":{"1min":8.9909999999999997,"5min":6.7830000000000004,"15min":6.5789999999999997},"max":{"1min":63.802999999999997,"5min":113.84,"15min":360.161},"last":34.058},{"interface":"front","average":{"1min":27.309999999999999,"5min":24.93,"15min":23.541},"min":{"1min":9.0510000000000002,"5min":9.0510000000000002,"15min":7.4409999999999998},"max":{"1min":114.33199999999999,"5min":226.68000000000001,"15min":364.39800000000002},"last":37.947000000000003}]},{"osd":5,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":39.484999999999999,"5min":28.780000000000001,"15min":29.396999999999998},"min":{"1min":9.6669999999999998,"5min":9.6669999999999998,"15min":7.4359999999999999},"max":{"1min":157.357,"5min":157.357,"15min":282.63900000000001},"last":99.200999999999993},{"interface":"front","average":{"1min":29.167999999999999,"5min":24.120999999999999,"15min":26.896000000000001},"min":{"1min":7.6630000000000003,"5min":7.2889999999999997,"15min":7.1529999999999996},"max":{"1min":105.211,"5min":163.33600000000001,"15min":382.01400000000001},"last":126.55}]},{"osd":6,"last update":"Sat Nov 25 05:25:21 2023","interfaces":[{"interface":"back","average":{"1min":18.364000000000001,"5min":24.588000000000001,"15min":47.482999999999997},"min":{"1min":10.593999999999999,"5min":7.5209999999999999,"15min":6.8230000000000004},"max":{"1min":64.594999999999999,"5min":187.62100000000001,"15min":3420.1689999999999},"last":32.476999999999997},{"interface":"front","average":{"1min":22.228999999999999,"5min":26.702000000000002,"15min":40.634999999999998},"min":{"1min":12.193,"5min":9.9260000000000002,"15min":9.423},"max":{"1min":70.814999999999998,"5min":131.99600000000001,"15min":1922.326},"last":34.700000000000003}]},{"osd":7,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":23.481999999999999,"5min":23.608000000000001,"15min":24.239999999999998},"min":{"1min":8.8559999999999999,"5min":7.0279999999999996,"15min":7.0279999999999996},"max":{"1min":116.39400000000001,"5min":219.57900000000001,"15min":272.03699999999998},"last":29.381},{"interface":"front","average":{"1min":30.045999999999999,"5min":26.843,"15min":27.969000000000001},"min":{"1min":11.65,"5min":10.239000000000001,"15min":8.3889999999999993},"max":{"1min":118.447,"5min":126.80200000000001,"15min":297.976},"last":35.962000000000003}]}]},{"osd":3,"up_from":16,"seq":68719477000,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84748,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94287092,"statfs":{"total":96636764160,"available":96549982208,"internally_reserved":0,"allocated":13598720,"data_stored":8282535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":21.867000000000001,"5min":16.334,"15min":21.956},"min":{"1min":5.556,"5min":4.3090000000000002,"15min":4.3090000000000002},"max":{"1min":100.45399999999999,"5min":124.46299999999999,"15min":394.84199999999998},"last":7.1159999999999997},{"interface":"front","average":{"1min":19.355,"5min":17.161000000000001,"15min":21.390999999999998},"min":{"1min":7.2969999999999997,"5min":4.9020000000000001,"15min":4.5890000000000004},"max":{"1min":84.197000000000003,"5min":119.611,"15min":379.51900000000001},"last":12.651}]},{"osd":1,"last update":"Sat Nov 25 05:26:14 2023","interfaces":[{"interface":"back","average":{"1min":21.181999999999999,"5min":20.585000000000001,"15min":28.129999999999999},"min":{"1min":7.6379999999999999,"5min":7.1779999999999999,"15min":5.4080000000000004},"max":{"1min":75.337999999999994,"5min":132.477,"15min":674.39599999999996},"last":11.920999999999999},{"interface":"front","average":{"1min":18.774000000000001,"5min":24.016999999999999,"15min":26.608000000000001},"min":{"1min":6.1829999999999998,"5min":6.1829999999999998,"15min":6.1829999999999998},"max":{"1min":62.155999999999999,"5min":253.03100000000001,"15min":399.98500000000001},"last":19.158999999999999}]},{"osd":2,"last update":"Sat Nov 25 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":18.132000000000001,"5min":16.486999999999998,"15min":34.378999999999998},"min":{"1min":7.798,"5min":7.2649999999999997,"15min":7.0620000000000003},"max":{"1min":83.954999999999998,"5min":83.954999999999998,"15min":673.12400000000002},"last":9.3879999999999999},{"interface":"front","average":{"1min":22.617000000000001,"5min":22.503,"15min":35.618000000000002},"min":{"1min":7.351,"5min":7.351,"15min":7.351},"max":{"1min":77.671999999999997,"5min":227.36199999999999,"15min":492.04899999999998},"last":14.470000000000001}]},{"osd":4,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":21.428999999999998,"5min":21.751999999999999,"15min":31.024000000000001},"min":{"1min":8.8360000000000003,"5min":8.8360000000000003,"15min":7.6890000000000001},"max":{"1min":81.578000000000003,"5min":141.38399999999999,"15min":676.21299999999997},"last":22.710000000000001},{"interface":"front","average":{"1min":19.236000000000001,"5min":20.013999999999999,"15min":23.495999999999999},"min":{"1min":7.6040000000000001,"5min":6.641,"15min":6.1559999999999997},"max":{"1min":77.034999999999997,"5min":313.18299999999999,"15min":313.18299999999999},"last":15.223000000000001}]},{"osd":5,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":26.818000000000001,"5min":23.085999999999999,"15min":32.587000000000003},"min":{"1min":10.710000000000001,"5min":9.6999999999999993,"15min":9.3360000000000003},"max":{"1min":80.661000000000001,"5min":194.51400000000001,"15min":675.37800000000004},"last":17.161000000000001},{"interface":"front","average":{"1min":27.657,"5min":21.199999999999999,"15min":28.454999999999998},"min":{"1min":11.975,"5min":7.5190000000000001,"15min":7.5190000000000001},"max":{"1min":103.021,"5min":131.768,"15min":314.40800000000002},"last":23.495000000000001}]},{"osd":6,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":23.952999999999999,"5min":20.739999999999998,"15min":24.882000000000001},"min":{"1min":8.5519999999999996,"5min":8.3030000000000008,"15min":7.0190000000000001},"max":{"1min":83.287999999999997,"5min":172.66,"15min":312.20699999999999},"last":13.022},{"interface":"front","average":{"1min":26.363,"5min":24.664000000000001,"15min":35.289999999999999},"min":{"1min":11.327,"5min":10.263999999999999,"15min":10.029},"max":{"1min":102.366,"5min":185.809,"15min":677.03499999999997},"last":21.332000000000001}]},{"osd":7,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":23.309000000000001,"5min":23.741,"15min":28.478999999999999},"min":{"1min":11.162000000000001,"5min":8.1609999999999996,"15min":7.6420000000000003},"max":{"1min":68.536000000000001,"5min":315.61500000000001,"15min":315.61500000000001},"last":15.903},{"interface":"front","average":{"1min":26.361999999999998,"5min":24.396000000000001,"15min":30.718},"min":{"1min":10.535,"5min":10.535,"15min":10.449999999999999},"max":{"1min":102.361,"5min":145.86099999999999,"15min":320.38400000000001},"last":25.646000000000001}]}]},{"osd":0,"up_from":16,"seq":68719477001,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66664,"kb_used_data":13256,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305176,"statfs":{"total":96636764160,"available":96568500224,"internally_reserved":0,"allocated":13574144,"data_stored":8268699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":12.904999999999999,"5min":15.397,"15min":17.724},"min":{"1min":4.6100000000000003,"5min":4.6100000000000003,"15min":4.6100000000000003},"max":{"1min":39.119,"5min":109.95399999999999,"15min":209.29900000000001},"last":23.010999999999999},{"interface":"front","average":{"1min":12.183999999999999,"5min":16.882000000000001,"15min":19.984999999999999},"min":{"1min":5.6219999999999999,"5min":4.5869999999999997,"15min":4.4420000000000002},"max":{"1min":56.362000000000002,"5min":119.73,"15min":401.36000000000001},"last":16.911000000000001}]},{"osd":2,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":17.683,"5min":19.295000000000002,"15min":22.402999999999999},"min":{"1min":6.6680000000000001,"5min":6.5380000000000003,"15min":6.1260000000000003},"max":{"1min":71.555999999999997,"5min":94.311999999999998,"15min":460.572},"last":35.844999999999999},{"interface":"front","average":{"1min":17.940000000000001,"5min":18.923999999999999,"15min":22.27},"min":{"1min":7.4980000000000002,"5min":6.5279999999999996,"15min":6.3090000000000002},"max":{"1min":73.290000000000006,"5min":89.322999999999993,"15min":405.94099999999997},"last":37.648000000000003}]},{"osd":3,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":21.096,"5min":23.585999999999999,"15min":24.260000000000002},"min":{"1min":7.2999999999999998,"5min":7.2999999999999998,"15min":6.8540000000000001},"max":{"1min":53.616999999999997,"5min":157.65299999999999,"15min":267.74599999999998},"last":29.195},{"interface":"front","average":{"1min":19.989000000000001,"5min":23.132000000000001,"15min":23.312000000000001},"min":{"1min":9.8480000000000008,"5min":7.1269999999999998,"15min":6.4080000000000004},"max":{"1min":55.122,"5min":128.14500000000001,"15min":266.81900000000002},"last":24.175999999999998}]},{"osd":4,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":21.091000000000001,"5min":20.103999999999999,"15min":20.606000000000002},"min":{"1min":9.4649999999999999,"5min":8.3940000000000001,"15min":6.2750000000000004},"max":{"1min":72.540999999999997,"5min":115.369,"15min":147.86099999999999},"last":11.446},{"interface":"front","average":{"1min":26.452000000000002,"5min":23.149999999999999,"15min":22.498999999999999},"min":{"1min":9.2189999999999994,"5min":6.8250000000000002,"15min":6.8250000000000002},"max":{"1min":110.93300000000001,"5min":127.291,"15min":148.96199999999999},"last":9.4199999999999999}]},{"osd":5,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.902999999999999,"5min":23.966999999999999,"15min":23.213999999999999},"min":{"1min":9.0109999999999992,"5min":6.9829999999999997,"15min":6.9829999999999997},"max":{"1min":69.957999999999998,"5min":184.80000000000001,"15min":184.80000000000001},"last":136},{"interface":"front","average":{"1min":20.024000000000001,"5min":23.408999999999999,"15min":25.248000000000001},"min":{"1min":10.836,"5min":7.3849999999999998,"15min":7.3849999999999998},"max":{"1min":58.210000000000001,"5min":131.583,"15min":395.65100000000001},"last":227.119}]},{"osd":6,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":21.329000000000001,"5min":23.550000000000001,"15min":24.617000000000001},"min":{"1min":9.2829999999999995,"5min":8.4809999999999999,"15min":8.4809999999999999},"max":{"1min":71.225999999999999,"5min":116.63800000000001,"15min":161.24299999999999},"last":18.061},{"interface":"front","average":{"1min":26.015999999999998,"5min":25.968,"15min":26.870000000000001},"min":{"1min":13.154999999999999,"5min":9.7759999999999998,"15min":9.7759999999999998},"max":{"1min":71.831999999999994,"5min":115.986,"15min":138.46799999999999},"last":15.566000000000001}]},{"osd":7,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.158000000000001,"5min":18.911000000000001,"15min":20.722999999999999},"min":{"1min":10.941000000000001,"5min":8.1340000000000003,"15min":7.1079999999999997},"max":{"1min":40.393999999999998,"5min":112.73099999999999,"15min":140.595},"last":21.469999999999999},{"interface":"front","average":{"1min":21.975999999999999,"5min":22.829999999999998,"15min":25.202999999999999},"min":{"1min":11.441000000000001,"5min":10.888999999999999,"15min":9.9860000000000007},"max":{"1min":57.088000000000001,"5min":103.19,"15min":339.298},"last":42.442999999999998}]}]},{"osd":6,"up_from":15,"seq":64424509704,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84768,"kb_used_data":13312,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94287072,"statfs":{"total":96636764160,"available":96549961728,"internally_reserved":0,"allocated":13631488,"data_stored":8340674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":16.198,"5min":17.219999999999999,"15min":19.709},"min":{"1min":4.7489999999999997,"5min":4.6740000000000004,"15min":4.4409999999999998},"max":{"1min":117.05500000000001,"5min":117.05500000000001,"15min":203.857},"last":25.263000000000002},{"interface":"front","average":{"1min":15.382999999999999,"5min":16.821000000000002,"15min":20.439},"min":{"1min":5.4820000000000002,"5min":5.1210000000000004,"15min":4.6289999999999996},"max":{"1min":117.795,"5min":236.78700000000001,"15min":381.43900000000002},"last":31.957999999999998}]},{"osd":1,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.943999999999999,"5min":22.879000000000001,"15min":26.055},"min":{"1min":8.0890000000000004,"5min":6.2839999999999998,"15min":6.165},"max":{"1min":143.61099999999999,"5min":229.53399999999999,"15min":310.423},"last":11.465999999999999},{"interface":"front","average":{"1min":20.885999999999999,"5min":20.399999999999999,"15min":26.382999999999999},"min":{"1min":6.3360000000000003,"5min":6.2030000000000003,"15min":5.1310000000000002},"max":{"1min":190.029,"5min":243.74700000000001,"15min":548.03599999999994},"last":5.9429999999999996}]},{"osd":2,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":28.033999999999999,"5min":22.832000000000001,"15min":27.013999999999999},"min":{"1min":7.9550000000000001,"5min":6.9690000000000003,"15min":6.9690000000000003},"max":{"1min":122.754,"5min":251.524,"15min":353.80399999999997},"last":90.819000000000003},{"interface":"front","average":{"1min":26.027000000000001,"5min":27.611999999999998,"15min":30.841999999999999},"min":{"1min":10.65,"5min":10.65,"15min":10.617000000000001},"max":{"1min":120.961,"5min":249.97399999999999,"15min":304.49900000000002},"last":90.048000000000002}]},{"osd":3,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":25.802,"5min":31.065999999999999,"15min":32.701999999999998},"min":{"1min":11.24,"5min":10.164,"15min":7.5419999999999998},"max":{"1min":185.28299999999999,"5min":259.89299999999997,"15min":295.82799999999997},"last":156.45099999999999},{"interface":"front","average":{"1min":22.247,"5min":26.661000000000001,"15min":26.797999999999998},"min":{"1min":7.7720000000000002,"5min":6.9199999999999999,"15min":6.0209999999999999},"max":{"1min":168.56100000000001,"5min":249.07400000000001,"15min":266.40199999999999},"last":181.51599999999999}]},{"osd":4,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":31.027999999999999,"5min":31.033000000000001,"15min":32.439999999999998},"min":{"1min":10.000999999999999,"5min":10.000999999999999,"15min":10.000999999999999},"max":{"1min":186.28,"5min":277.85000000000002,"15min":330.86900000000003},"last":12.423},{"interface":"front","average":{"1min":25.614999999999998,"5min":26.050000000000001,"15min":25.638000000000002},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":6.5919999999999996},"max":{"1min":192.25999999999999,"5min":288.13099999999997,"15min":288.13099999999997},"last":8.3740000000000006}]},{"osd":5,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.140000000000001,"5min":18.035,"15min":22.939},"min":{"1min":6.0039999999999996,"5min":5.5590000000000002,"15min":5.5590000000000002},"max":{"1min":121.31399999999999,"5min":237.46100000000001,"15min":281.13999999999999},"last":7.5819999999999999},{"interface":"front","average":{"1min":25.876000000000001,"5min":28.905999999999999,"15min":33.758000000000003},"min":{"1min":8.0069999999999997,"5min":8.0069999999999997,"15min":8.0069999999999997},"max":{"1min":133.44999999999999,"5min":269.77100000000002,"15min":311.09199999999998},"last":13.314}]},{"osd":7,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":29.199999999999999,"5min":31.532,"15min":34.887999999999998},"min":{"1min":13.699,"5min":12.319000000000001,"15min":9.2210000000000001},"max":{"1min":140.25200000000001,"5min":259.87700000000001,"15min":367.24900000000002},"last":14.119},{"interface":"front","average":{"1min":25.48,"5min":24.911999999999999,"15min":28.138999999999999},"min":{"1min":8.8529999999999998,"5min":8.3469999999999995,"15min":7.7199999999999998},"max":{"1min":161.37899999999999,"5min":269.14400000000001,"15min":325.54199999999997},"last":9.8339999999999996}]}]},{"osd":4,"up_from":15,"seq":64424509706,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70816,"kb_used_data":13260,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94301024,"statfs":{"total":96636764160,"available":96564248576,"internally_reserved":0,"allocated":13578240,"data_stored":8268704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":17.417999999999999,"5min":18.896000000000001,"15min":24.765999999999998},"min":{"1min":5.0439999999999996,"5min":4.3170000000000002,"15min":3.8279999999999998},"max":{"1min":103.83199999999999,"5min":150.73599999999999,"15min":361.67000000000002},"last":8.6720000000000006},{"interface":"front","average":{"1min":12.904,"5min":17.263999999999999,"15min":23.478999999999999},"min":{"1min":5.3449999999999998,"5min":4.5369999999999999,"15min":4.1289999999999996},"max":{"1min":55.915999999999997,"5min":216.86600000000001,"15min":270.81},"last":8.2230000000000008}]},{"osd":1,"last update":"Sat Nov 25 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":12.121,"5min":19.879999999999999,"15min":24.692},"min":{"1min":5.2149999999999999,"5min":5.0519999999999996,"15min":4.9749999999999996},"max":{"1min":58.972000000000001,"5min":205.36500000000001,"15min":445.61500000000001},"last":11.268000000000001},{"interface":"front","average":{"1min":15.630000000000001,"5min":20.486000000000001,"15min":24.024000000000001},"min":{"1min":10.656000000000001,"5min":6.5759999999999996,"15min":6.5759999999999996},"max":{"1min":30.831,"5min":155.90299999999999,"15min":274.81099999999998},"last":13.839}]},{"osd":2,"last update":"Sat Nov 25 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":15.132,"5min":20.442,"15min":28.692},"min":{"1min":9.8219999999999992,"5min":6.9320000000000004,"15min":6.9320000000000004},"max":{"1min":31.68,"5min":170.72200000000001,"15min":334.29399999999998},"last":12.66},{"interface":"front","average":{"1min":21.068000000000001,"5min":24.486000000000001,"15min":31.969999999999999},"min":{"1min":11.401,"5min":10.664,"15min":8.5899999999999999},"max":{"1min":74.477000000000004,"5min":165.952,"15min":348.54599999999999},"last":15.205}]},{"osd":3,"last update":"Sat Nov 25 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":14.75,"5min":24.774000000000001,"15min":37.012},"min":{"1min":8.0649999999999995,"5min":6.5810000000000004,"15min":6.133},"max":{"1min":40.886000000000003,"5min":206.09899999999999,"15min":588.27800000000002},"last":13.148999999999999},{"interface":"front","average":{"1min":16.039000000000001,"5min":25.753,"15min":28.155000000000001},"min":{"1min":6.9210000000000003,"5min":6.4740000000000002,"15min":6.4740000000000002},"max":{"1min":75.966999999999999,"5min":204.739,"15min":381.53100000000001},"last":6.54}]},{"osd":5,"last update":"Sat Nov 25 05:25:56 2023","interfaces":[{"interface":"back","average":{"1min":18.43,"5min":20.623999999999999,"15min":26.562000000000001},"min":{"1min":5.875,"5min":5.875,"15min":5.875},"max":{"1min":72.236999999999995,"5min":195.851,"15min":360.387},"last":16.718},{"interface":"front","average":{"1min":24.617000000000001,"5min":24.105,"15min":29.986999999999998},"min":{"1min":12.101000000000001,"5min":9.4469999999999992,"15min":9.4469999999999992},"max":{"1min":122.352,"5min":159.67099999999999,"15min":274.38200000000001},"last":18.823}]},{"osd":6,"last update":"Sat Nov 25 05:25:56 2023","interfaces":[{"interface":"back","average":{"1min":20.521000000000001,"5min":24.015000000000001,"15min":27.704999999999998},"min":{"1min":10.093,"5min":10.093,"15min":9.8409999999999993},"max":{"1min":48.161000000000001,"5min":169.44499999999999,"15min":181.43299999999999},"last":17.812999999999999},{"interface":"front","average":{"1min":24.329000000000001,"5min":22.428999999999998,"15min":23.622},"min":{"1min":15.603999999999999,"5min":7.1619999999999999,"15min":5.6390000000000002},"max":{"1min":60.040999999999997,"5min":212.25399999999999,"15min":219.26499999999999},"last":9.7159999999999993}]},{"osd":7,"last update":"Sat Nov 25 05:25:56 2023","interfaces":[{"interface":"back","average":{"1min":14.73,"5min":23.181000000000001,"15min":25.620999999999999},"min":{"1min":8.7100000000000009,"5min":7.0369999999999999,"15min":7.0369999999999999},"max":{"1min":27.728999999999999,"5min":202.363,"15min":215.512},"last":10.627000000000001},{"interface":"front","average":{"1min":14.417999999999999,"5min":20.943999999999999,"15min":24.698},"min":{"1min":6.0350000000000001,"5min":6.0350000000000001,"15min":6.0350000000000001},"max":{"1min":29.920999999999999,"5min":211.62200000000001,"15min":248.83199999999999},"last":14.505000000000001}]}]},{"osd":7,"up_from":13,"seq":55834575112,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66668,"kb_used_data":13272,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305172,"statfs":{"total":96636764160,"available":96568496128,"internally_reserved":0,"allocated":13590528,"data_stored":8295623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":11.608000000000001,"5min":13.819000000000001,"15min":17.303999999999998},"min":{"1min":4.6150000000000002,"5min":4.3579999999999997,"15min":4.3579999999999997},"max":{"1min":23.535,"5min":97.569000000000003,"15min":222.50800000000001},"last":28.766999999999999},{"interface":"front","average":{"1min":9.8520000000000003,"5min":15.375999999999999,"15min":18.193999999999999},"min":{"1min":4.7450000000000001,"5min":4.6600000000000001,"15min":4.5579999999999998},"max":{"1min":25.248000000000001,"5min":218.03999999999999,"15min":218.03999999999999},"last":34.869999999999997}]},{"osd":1,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.643000000000001,"5min":13.715999999999999,"15min":17.489000000000001},"min":{"1min":5.4859999999999998,"5min":5.4859999999999998,"15min":5.2619999999999996},"max":{"1min":79.078999999999994,"5min":79.078999999999994,"15min":165.99000000000001},"last":8.4079999999999995},{"interface":"front","average":{"1min":20.498000000000001,"5min":18.827000000000002,"15min":22.771999999999998},"min":{"1min":7.508,"5min":6.8140000000000001,"15min":6.7030000000000003},"max":{"1min":117.25,"5min":117.25,"15min":303.892},"last":10.375}]},{"osd":2,"last update":"Sat Nov 25 05:25:16 2023","interfaces":[{"interface":"back","average":{"1min":16.558,"5min":20.265000000000001,"15min":24.670999999999999},"min":{"1min":8.1530000000000005,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":26.408000000000001,"5min":186.08600000000001,"15min":214.37100000000001},"last":21.869},{"interface":"front","average":{"1min":13.426,"5min":19.419,"15min":23.486999999999998},"min":{"1min":7.8540000000000001,"5min":7.508,"15min":6.4279999999999999},"max":{"1min":23.759,"5min":192.584,"15min":337.41899999999998},"last":22.702000000000002}]},{"osd":3,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":16.559999999999999,"5min":22.437999999999999,"15min":23.844999999999999},"min":{"1min":12.122999999999999,"5min":8.4469999999999992,"15min":7.5199999999999996},"max":{"1min":28.463000000000001,"5min":152.47399999999999,"15min":238.941},"last":13.661},{"interface":"front","average":{"1min":18.123000000000001,"5min":21.559999999999999,"15min":26.684999999999999},"min":{"1min":12.221,"5min":6.6929999999999996,"15min":5.5949999999999998},"max":{"1min":29.536000000000001,"5min":150.208,"15min":1292.413},"last":17.611999999999998}]},{"osd":4,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.506,"5min":23.238,"15min":25.268000000000001},"min":{"1min":8.8109999999999999,"5min":8.8109999999999999,"15min":7.9340000000000002},"max":{"1min":21.449999999999999,"5min":120.414,"15min":328.31400000000002},"last":10.839},{"interface":"front","average":{"1min":27.879000000000001,"5min":21.405000000000001,"15min":23.079999999999998},"min":{"1min":10.739000000000001,"5min":7.1849999999999996,"15min":6.4320000000000004},"max":{"1min":192.684,"5min":192.684,"15min":302.97199999999998},"last":27.952000000000002}]},{"osd":5,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":17.332000000000001,"5min":22.279,"15min":24.291},"min":{"1min":9.1509999999999998,"5min":7.7919999999999998,"15min":7.7619999999999996},"max":{"1min":42.118000000000002,"5min":186.69499999999999,"15min":217.023},"last":21.094000000000001},{"interface":"front","average":{"1min":23.495000000000001,"5min":28.128,"15min":29.045999999999999},"min":{"1min":11.694000000000001,"5min":10.643000000000001,"15min":7.1130000000000004},"max":{"1min":83.798000000000002,"5min":210.22300000000001,"15min":318.11599999999999},"last":30.222000000000001}]},{"osd":6,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.75,"5min":20.798999999999999,"15min":22.933},"min":{"1min":7.069,"5min":6.2380000000000004,"15min":6.2380000000000004},"max":{"1min":100.828,"5min":188.15199999999999,"15min":282.28100000000001},"last":26.225999999999999},{"interface":"front","average":{"1min":18.027999999999999,"5min":23.329000000000001,"15min":25.382000000000001},"min":{"1min":8.0459999999999994,"5min":8.0459999999999994,"15min":8.0459999999999994},"max":{"1min":100.143,"5min":187.09899999999999,"15min":215.00700000000001},"last":47.826000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575112,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89288,"kb_used_data":13672,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94282552,"statfs":{"total":96636764160,"available":96545333248,"internally_reserved":0,"allocated":14000128,"data_stored":8680375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":78,"apply_latency_ms":78,"commit_latency_ns":78000000,"apply_latency_ns":78000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":17.623000000000001,"5min":14.541,"15min":18.605},"min":{"1min":4.5910000000000002,"5min":4.202,"15min":4.202},"max":{"1min":133.92599999999999,"5min":167.83600000000001,"15min":471.81},"last":29.609000000000002},{"interface":"front","average":{"1min":14.865,"5min":15.755000000000001,"15min":20.606999999999999},"min":{"1min":6.2699999999999996,"5min":5.4279999999999999,"15min":4.5529999999999999},"max":{"1min":46.905000000000001,"5min":197.834,"15min":317.45699999999999},"last":9.3529999999999998}]},{"osd":1,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":27.741,"5min":18.356000000000002,"15min":21.565000000000001},"min":{"1min":6.3470000000000004,"5min":5.4189999999999996,"15min":5.0709999999999997},"max":{"1min":140.28999999999999,"5min":140.28999999999999,"15min":263.589},"last":8.5440000000000005},{"interface":"front","average":{"1min":30.024000000000001,"5min":17.751000000000001,"15min":23.059999999999999},"min":{"1min":6.351,"5min":5.1459999999999999,"15min":5.1459999999999999},"max":{"1min":150.196,"5min":150.196,"15min":238.26499999999999},"last":10.954000000000001}]},{"osd":2,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":19.062999999999999,"5min":19.102,"15min":23.210000000000001},"min":{"1min":7.1890000000000001,"5min":6.2240000000000002,"15min":6.2240000000000002},"max":{"1min":82.295000000000002,"5min":182.477,"15min":416.83199999999999},"last":13.824999999999999},{"interface":"front","average":{"1min":28.707000000000001,"5min":25.138000000000002,"15min":25.091999999999999},"min":{"1min":12.006,"5min":9.4000000000000004,"15min":7.0880000000000001},"max":{"1min":179.749,"5min":190.517,"15min":207.28700000000001},"last":14.474}]},{"osd":3,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":31.044,"5min":24.114000000000001,"15min":29.751999999999999},"min":{"1min":10.327,"5min":10.097,"15min":6.6079999999999997},"max":{"1min":166.59700000000001,"5min":166.59700000000001,"15min":443.10300000000001},"last":12.470000000000001},{"interface":"front","average":{"1min":28.024999999999999,"5min":19.308,"15min":31.902000000000001},"min":{"1min":7.851,"5min":6.9800000000000004,"15min":6.2199999999999998},"max":{"1min":130.87299999999999,"5min":130.87299999999999,"15min":1666.6489999999999},"last":35.137999999999998}]},{"osd":4,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":30.039000000000001,"5min":25.219999999999999,"15min":25.515999999999998},"min":{"1min":10.957000000000001,"5min":10.657,"15min":9.0280000000000005},"max":{"1min":173.041,"5min":173.041,"15min":173.041},"last":11.816000000000001},{"interface":"front","average":{"1min":17.225999999999999,"5min":17.971,"15min":21.449000000000002},"min":{"1min":6.5330000000000004,"5min":6.5330000000000004,"15min":6.5330000000000004},"max":{"1min":68.790000000000006,"5min":123.01300000000001,"15min":156.92500000000001},"last":32.195999999999998}]},{"osd":6,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":26.797999999999998,"5min":26.663,"15min":31.244},"min":{"1min":13.439,"5min":11.666,"15min":9.9979999999999993},"max":{"1min":100.47799999999999,"5min":229.86199999999999,"15min":229.86199999999999},"last":15.119999999999999},{"interface":"front","average":{"1min":31.841999999999999,"5min":24.629000000000001,"15min":27.375},"min":{"1min":8.6280000000000001,"5min":8.1159999999999997,"15min":7.4770000000000003},"max":{"1min":165.46000000000001,"5min":234.31100000000001,"15min":234.31100000000001},"last":33.174999999999997}]},{"osd":7,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":25.68,"5min":20.594999999999999,"15min":26.533999999999999},"min":{"1min":7.2679999999999998,"5min":6.2370000000000001,"15min":6.2370000000000001},"max":{"1min":67.870999999999995,"5min":195.94399999999999,"15min":195.94399999999999},"last":13.129},{"interface":"front","average":{"1min":23.308,"5min":25.666,"15min":26.741},"min":{"1min":11.151,"5min":11.151,"15min":7.3289999999999997},"max":{"1min":57.329000000000001,"5min":185.74700000000001,"15min":224.99600000000001},"last":34.164000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-25T05:26:17.824 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-25T05:26:17.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-25T05:26:17.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.928+0000 7f2aa1547640 10 monclient: get_monmap_and_config 2023-11-25T05:26:17.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.928+0000 7f2aa1547640 10 monclient: build_initial_monmap 2023-11-25T05:26:17.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.928+0000 7f2aa1547640 10 monclient: monmap: 2023-11-25T05:26:17.930 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:17.930 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:17.930 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:17.929610+0000 2023-11-25T05:26:17.931 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:17.929610+0000 2023-11-25T05:26:17.931 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:17.931 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:17.931 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:17.931 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:17.931 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:17.931 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:17.931 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:17.931 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:17.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 1 Processor -- start 2023-11-25T05:26:17.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 1 -- start start 2023-11-25T05:26:17.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 10 monclient: init 2023-11-25T05:26:17.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:17.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-11-25T05:26:17.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a9c0a8950 0x7f2a9c13f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:17.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 10 monclient(hunting): picked mon.noname-a con 0x7f2a9c0a8950 addr v2:172.21.15.7:3300/0 2023-11-25T05:26:17.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a9c0a8d30 addr v1:172.21.15.7:6789/0 2023-11-25T05:26:17.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2a9c145ec0 0x7f2a9c13f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:17.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 10 monclient(hunting): picked mon.noname-c con 0x7f2a9c145ec0 addr v2:172.21.15.90:3300/0 2023-11-25T05:26:17.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c145320 con 0x7f2a9c0a8d30 2023-11-25T05:26:17.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:17.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a9c0b2260 con 0x7f2a9c0a8950 2023-11-25T05:26:17.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:17.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a9c0aa5e0 con 0x7f2a9c145ec0 2023-11-25T05:26:17.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:17.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.929+0000 7f2aa1547640 10 monclient(hunting): authenticate will time out at 2704.172761s 2023-11-25T05:26:17.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.930+0000 7f2a9affd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a9c0a8950 0x7f2a9c13f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:17.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.930+0000 7f2a9b7fe640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2a9c145ec0 0x7f2a9c13f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:17.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9b7fe640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2a9c145ec0 0x7f2a9c13f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:39212/0 (socket says 172.21.15.7:39212) 2023-11-25T05:26:17.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9b7fe640 1 -- 172.21.15.7:0/3139525558 learned_addr learned my addr 172.21.15.7:0/3139525558 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:17.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9b7fe640 10 monclient(hunting): get_auth_request con 0x7f2a9c145ec0 auth_method 0 2023-11-25T05:26:17.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:17.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9b7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:17.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:17.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9affd640 10 monclient(hunting): get_auth_request con 0x7f2a9c0a8950 auth_method 0 2023-11-25T05:26:17.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:17.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9affd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:17.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9affd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:17.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:17.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:17.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:17.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:17.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:17.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.931+0000 7f2a9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:17.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.933+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_done global_id 21217 payload 274 2023-11-25T05:26:17.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.933+0000 7f2a9b7fe640 1 -- 172.21.15.7:0/3139525558 >> v2:172.21.15.7:3300/0 conn(0x7f2a9c0a8950 msgr2=0x7f2a9c13f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:17.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.933+0000 7f2a9b7fe640 1 --2- 172.21.15.7:0/3139525558 >> v2:172.21.15.7:3300/0 conn(0x7f2a9c0a8950 0x7f2a9c13f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.933+0000 7f2a9b7fe640 1 -- 172.21.15.7:0/3139525558 >> v1:172.21.15.7:6789/0 conn(0x7f2a9c0a8d30 legacy=0x7f2a9c1457b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:17.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.933+0000 7f2a9b7fe640 10 monclient: _finish_hunting 0 2023-11-25T05:26:17.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.933+0000 7f2a9b7fe640 1 monclient: found mon.noname-c 2023-11-25T05:26:17.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.933+0000 7f2a9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:17.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.933+0000 7f2a9b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:26:17.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.933+0000 7f2a9b7fe640 1 -- 172.21.15.7:0/3139525558 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c0aa8c0 con 0x7f2a9c145ec0 2023-11-25T05:26:17.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.935+0000 7f2a9b7fe640 1 --2- 172.21.15.7:0/3139525558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a9c145ec0 0x7f2a9c13f6d0 secure :-1 s=READY pgs=5145 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0b8550 tx=0x7f2a9402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f69f0544a9b17cd server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2a9a7fc640 1 -- 172.21.15.7:0/3139525558 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9402ca80 con 0x7f2a9c145ec0 2023-11-25T05:26:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2a9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2a9a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:26:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2a9a7fc640 10 monclient: dump: 2023-11-25T05:26:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:17.941 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2a9a7fc640 10 monclient: _finish_auth 0 2023-11-25T05:26:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2a9a7fc640 10 monclient: _check_auth_tickets 2023-11-25T05:26:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2a9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2a9a7fc640 1 -- 172.21.15.7:0/3139525558 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9402cbe0 con 0x7f2a9c145ec0 2023-11-25T05:26:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2aa1547640 5 monclient: authenticate success, global_id 21217 2023-11-25T05:26:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2aa1547640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:26:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2a9a7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2aa1547640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2a9a7fc640 1 -- 172.21.15.7:0/3139525558 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a940343f0 con 0x7f2a9c145ec0 2023-11-25T05:26:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2a9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2aa1547640 4 set_mon_vals no callback set 2023-11-25T05:26:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2a9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.938+0000 7f2a9a7fc640 10 monclient: dump: 2023-11-25T05:26:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:17.943 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:17.943 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:17.943 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:17.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.939+0000 7f2aa1547640 10 monclient: shutdown 2023-11-25T05:26:17.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.939+0000 7f2aa1547640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:17.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.939+0000 7f2aa1547640 1 -- 172.21.15.7:0/3139525558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a9c145ec0 msgr2=0x7f2a9c13f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:17.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.939+0000 7f2aa1547640 1 --2- 172.21.15.7:0/3139525558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a9c145ec0 0x7f2a9c13f6d0 secure :-1 s=READY pgs=5145 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0b8550 tx=0x7f2a9402bee0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.939+0000 7f2aa1547640 1 -- 172.21.15.7:0/3139525558 shutdown_connections 2023-11-25T05:26:17.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.939+0000 7f2aa1547640 1 --2- 172.21.15.7:0/3139525558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a9c145ec0 0x7f2a9c13f6d0 unknown :-1 s=CLOSED pgs=5145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.939+0000 7f2aa1547640 1 --2- 172.21.15.7:0/3139525558 >> v2:172.21.15.7:3300/0 conn(0x7f2a9c0a8950 0x7f2a9c13f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.939+0000 7f2aa1547640 1 -- 172.21.15.7:0/3139525558 >> 172.21.15.7:0/3139525558 conn(0x7f2a9c061970 msgr2=0x7f2a9c0b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:17.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.939+0000 7f2aa1547640 1 -- 172.21.15.7:0/3139525558 shutdown_connections 2023-11-25T05:26:17.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.939+0000 7f2aa1547640 1 -- 172.21.15.7:0/3139525558 wait complete. 2023-11-25T05:26:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.940+0000 7f2aa1547640 10 monclient: build_initial_monmap 2023-11-25T05:26:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.940+0000 7f2aa1547640 10 monclient: monmap: 2023-11-25T05:26:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:17.948 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.940+0000 7f2aa1547640 1 librados: starting msgr at 2023-11-25T05:26:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.940+0000 7f2aa1547640 1 librados: starting objecter 2023-11-25T05:26:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.940+0000 7f2aa1547640 1 Processor -- start 2023-11-25T05:26:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.940+0000 7f2aa1547640 1 -- start start 2023-11-25T05:26:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.940+0000 7f2aa1547640 1 librados: setting wanted keys 2023-11-25T05:26:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.940+0000 7f2aa1547640 1 librados: calling monclient init 2023-11-25T05:26:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.940+0000 7f2aa1547640 10 monclient: init 2023-11-25T05:26:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-25T05:26:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a9c0a8950 0x7f2a9c13fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 10 monclient(hunting): picked mon.noname-b con 0x7f2a9c0a8950 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a9c0a8d30 0x7f2a9c1449f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 10 monclient(hunting): picked mon.noname-a con 0x7f2a9c0a8d30 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2a9c145ec0 0x7f2a9c133d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 10 monclient(hunting): picked mon.noname-c con 0x7f2a9c145ec0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c136f10 con 0x7f2a9c0a8d30 2023-11-25T05:26:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c135450 con 0x7f2a9c0a8950 2023-11-25T05:26:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c1330d0 con 0x7f2a9c145ec0 2023-11-25T05:26:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.941+0000 7f2aa1547640 10 monclient(hunting): authenticate will time out at 2704.184215s 2023-11-25T05:26:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a9b7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2a9c145ec0 0x7f2a9c133d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a927fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a9c0a8d30 0x7f2a9c1449f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a9affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a9c0a8950 0x7f2a9c13fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a9b7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2a9c145ec0 0x7f2a9c133d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38190/0 (socket says 172.21.15.7:38190) 2023-11-25T05:26:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a9b7fe640 1 -- 172.21.15.7:0/555123545 learned_addr learned my addr 172.21.15.7:0/555123545 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a9b7fe640 10 monclient(hunting): get_auth_request con 0x7f2a9c145ec0 auth_method 0 2023-11-25T05:26:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a9b7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a9b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a927fc640 10 monclient(hunting): get_auth_request con 0x7f2a9c0a8d30 auth_method 0 2023-11-25T05:26:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a927fc640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a927fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a9affd640 10 monclient(hunting): get_auth_request con 0x7f2a9c0a8950 auth_method 0 2023-11-25T05:26:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a9affd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a9affd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.942+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.943+0000 7f2a927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.943+0000 7f2a927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.943+0000 7f2a927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.943+0000 7f2a9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.943+0000 7f2a9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.943+0000 7f2a9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:17.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.945+0000 7f2a9affd640 10 monclient(hunting): handle_auth_done global_id 21220 payload 824 2023-11-25T05:26:17.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.945+0000 7f2a9affd640 1 -- 172.21.15.7:0/555123545 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2a9c145ec0 msgr2=0x7f2a9c133d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:17.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.945+0000 7f2a9affd640 1 --2- 172.21.15.7:0/555123545 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2a9c145ec0 0x7f2a9c133d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.945+0000 7f2a9affd640 1 -- 172.21.15.7:0/555123545 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a9c0a8d30 msgr2=0x7f2a9c1449f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:17.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.945+0000 7f2a9affd640 1 --2- 172.21.15.7:0/555123545 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a9c0a8d30 0x7f2a9c1449f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:17.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.945+0000 7f2a9affd640 10 monclient: _finish_hunting 0 2023-11-25T05:26:17.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.945+0000 7f2a9affd640 1 monclient: found mon.noname-b 2023-11-25T05:26:17.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.945+0000 7f2a9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:17.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.945+0000 7f2a9affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:26:17.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.945+0000 7f2a9affd640 1 -- 172.21.15.7:0/555123545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a94009b30 con 0x7f2a9c0a8950 2023-11-25T05:26:17.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.947+0000 7f2a9affd640 1 --2- 172.21.15.7:0/555123545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a9c0a8950 0x7f2a9c13fc20 secure :-1 s=READY pgs=5146 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00e850 tx=0x7f2a8c00ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:17.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.947+0000 7f2a93fff640 10 client.?.objecter ms_handle_connect 0x7f2a9c0a8950 2023-11-25T05:26:17.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.947+0000 7f2a93fff640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:17.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2a93fff640 1 -- 172.21.15.7:0/555123545 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c0095b0 con 0x7f2a9c0a8950 2023-11-25T05:26:17.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2a93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:17.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2a93fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:17.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2a93fff640 10 monclient: dump: 2023-11-25T05:26:17.957 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:17.957 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:17.957 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.957 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.957 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:17.957 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:17.957 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:17.957 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:17.957 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:17.957 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:17.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2a93fff640 10 monclient: _finish_auth 0 2023-11-25T05:26:17.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2a93fff640 10 monclient: _check_auth_tickets 2023-11-25T05:26:17.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2a93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:17.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2a93fff640 1 -- 172.21.15.7:0/555123545 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8c00c940 con 0x7f2a9c0a8950 2023-11-25T05:26:17.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2a93fff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:17.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2a93fff640 1 -- 172.21.15.7:0/555123545 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c00ccd0 con 0x7f2a9c0a8950 2023-11-25T05:26:17.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2a93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:17.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2a93fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:17.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2a93fff640 10 monclient: dump: 2023-11-25T05:26:17.958 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:17.958 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:17.958 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.958 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:17.958 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:17.959 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:17.959 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:17.959 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:17.959 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:17.959 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:17.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2aa1547640 5 monclient: authenticate success, global_id 21220 2023-11-25T05:26:17.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2aa1547640 10 monclient: _renew_subs 2023-11-25T05:26:17.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2aa1547640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:17.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2aa1547640 1 -- 172.21.15.7:0/555123545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9c144f30 con 0x7f2a9c0a8950 2023-11-25T05:26:17.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2a99ffb640 4 set_mon_vals no callback set 2023-11-25T05:26:17.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2aa1547640 10 client.21220.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:17.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2aa1547640 10 monclient: _renew_subs 2023-11-25T05:26:17.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2aa1547640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:17.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2aa1547640 1 -- 172.21.15.7:0/555123545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9c136a80 con 0x7f2a9c0a8950 2023-11-25T05:26:17.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.951+0000 7f2aa1547640 1 librados: init done 2023-11-25T05:26:17.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.952+0000 7f2aa1547640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:17.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.952+0000 7f2aa1547640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:17.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.952+0000 7f2aa1547640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:17.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.952+0000 7f2aa1547640 1 -- 172.21.15.7:0/555123545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a9c136d60 con 0x7f2a9c0a8950 2023-11-25T05:26:17.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.970+0000 7f2a93fff640 1 -- 172.21.15.7:0/555123545 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2a8c013470 con 0x7f2a9c0a8950 2023-11-25T05:26:17.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.970+0000 7f2a93fff640 1 --2- 172.21.15.7:0/555123545 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f2a7c06c7d0 0x7f2a7c06ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:17.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.970+0000 7f2a927fc640 1 --2- 172.21.15.7:0/555123545 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f2a7c06c7d0 0x7f2a7c06ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:17.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.971+0000 7f2a927fc640 10 monclient: get_auth_request con 0x7f2a7c06c7d0 auth_method 0 2023-11-25T05:26:17.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.971+0000 7f2a927fc640 1 --2- 172.21.15.7:0/555123545 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f2a7c06c7d0 0x7f2a7c06ec90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c142e20 tx=0x7f2a84007e30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:17.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.971+0000 7f2a93fff640 10 client.21220.objecter ms_handle_connect 0x7f2a7c06c7d0 2023-11-25T05:26:17.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.972+0000 7f2a93fff640 1 -- 172.21.15.7:0/555123545 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 1..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f2a8c00ce30 con 0x7f2a9c0a8950 2023-11-25T05:26:17.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.972+0000 7f2a93fff640 10 client.21220.objecter ms_dispatch 0x7f2a9c000b90 osd_map(918..918 src has 1..918) v4 2023-11-25T05:26:17.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.972+0000 7f2a93fff640 3 client.21220.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:17.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.972+0000 7f2a93fff640 3 client.21220.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:17.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:17.972+0000 7f2a93fff640 20 client.21220.objecter dump_active .. 0 homeless 2023-11-25T05:26:18.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.201+0000 7f2a93fff640 1 -- 172.21.15.7:0/555123545 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2a8c055090 con 0x7f2a9c0a8950 2023-11-25T05:26:18.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.201+0000 7f2a93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:18.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.201+0000 7f2a93fff640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:26:18.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.421+0000 7f2aa1547640 1 -- 172.21.15.7:0/555123545 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2a9c0aae80 con 0x7f2a7c06c7d0 2023-11-25T05:26:18.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.426+0000 7f2a93fff640 1 -- 172.21.15.7:0/555123545 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152053 (secure 0 0 0) 0x7f2a84003940 con 0x7f2a7c06c7d0 2023-11-25T05:26:18.429 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:26:18.429 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-25T05:26:18.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.428+0000 7f2aa1547640 10 librados: watch_flush enter 2023-11-25T05:26:18.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.428+0000 7f2aa1547640 10 librados: watch_flush exit 2023-11-25T05:26:18.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.428+0000 7f2aa1547640 20 client.21220.objecter shutdown clearing up homeless session... 2023-11-25T05:26:18.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.428+0000 7f2aa1547640 10 client.21220.objecter successfully canceled tick 2023-11-25T05:26:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.429+0000 7f2aa1547640 1 -- 172.21.15.7:0/555123545 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f2a7c06c7d0 msgr2=0x7f2a7c06ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.429+0000 7f2aa1547640 1 --2- 172.21.15.7:0/555123545 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f2a7c06c7d0 0x7f2a7c06ec90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c142e20 tx=0x7f2a84007e30 comp rx=0 tx=0).stop 2023-11-25T05:26:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.429+0000 7f2aa1547640 10 monclient: shutdown 2023-11-25T05:26:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.429+0000 7f2aa1547640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.429+0000 7f2aa1547640 1 -- 172.21.15.7:0/555123545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a9c0a8950 msgr2=0x7f2a9c13fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.429+0000 7f2aa1547640 1 --2- 172.21.15.7:0/555123545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a9c0a8950 0x7f2a9c13fc20 secure :-1 s=READY pgs=5146 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00e850 tx=0x7f2a8c00ed10 comp rx=0 tx=0).stop 2023-11-25T05:26:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.429+0000 7f2aa1547640 1 -- 172.21.15.7:0/555123545 shutdown_connections 2023-11-25T05:26:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.429+0000 7f2aa1547640 1 --2- 172.21.15.7:0/555123545 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f2a7c06c7d0 0x7f2a7c06ec90 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.429+0000 7f2aa1547640 1 --2- 172.21.15.7:0/555123545 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2a9c145ec0 0x7f2a9c133d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.429+0000 7f2aa1547640 1 --2- 172.21.15.7:0/555123545 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a9c0a8d30 0x7f2a9c1449f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.429+0000 7f2aa1547640 1 --2- 172.21.15.7:0/555123545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a9c0a8950 0x7f2a9c13fc20 unknown :-1 s=CLOSED pgs=5146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.429+0000 7f2aa1547640 1 -- 172.21.15.7:0/555123545 >> 172.21.15.7:0/555123545 conn(0x7f2a9c061970 msgr2=0x7f2a9c130aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.429+0000 7f2aa1547640 1 -- 172.21.15.7:0/555123545 shutdown_connections 2023-11-25T05:26:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.429+0000 7f2aa1547640 1 -- 172.21.15.7:0/555123545 wait complete. 2023-11-25T05:26:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.429+0000 7f2aa1547640 1 librados: shutdown 2023-11-25T05:26:18.441 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1580,"stamp":"2023-11-25T05:26:17.713644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":139,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":596648,"kb_used_data":106944,"kb_used_omap":12,"kb_used_meta":489459,"kb_avail":754378072,"statfs":{"total":773094113280,"available":772483145728,"internally_reserved":0,"allocated":109510656,"data_stored":67065247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":501206579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":389,"apply_latency_ms":389,"commit_latency_ns":389000000,"apply_latency_ns":389000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.158866"},"pg_stats":[{"pgid":"9.15","version":"0'0","reported_seq":861,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.913594+0000","last_change":"2023-11-25T05:05:46.884752+0000","last_active":"2023-11-25T05:26:15.913594+0000","last_peered":"2023-11-25T05:26:15.913594+0000","last_clean":"2023-11-25T05:26:15.913594+0000","last_became_active":"2023-11-25T05:05:46.875394+0000","last_became_peered":"2023-11-25T05:05:46.875394+0000","last_unstale":"2023-11-25T05:26:15.913594+0000","last_undegraded":"2023-11-25T05:26:15.913594+0000","last_fullsized":"2023-11-25T05:26:15.913594+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:08:59.865717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":870,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.130927+0000","last_change":"2023-11-25T05:05:46.929874+0000","last_active":"2023-11-25T05:26:16.130927+0000","last_peered":"2023-11-25T05:26:16.130927+0000","last_clean":"2023-11-25T05:26:16.130927+0000","last_became_active":"2023-11-25T05:05:46.923663+0000","last_became_peered":"2023-11-25T05:05:46.923663+0000","last_unstale":"2023-11-25T05:26:16.130927+0000","last_undegraded":"2023-11-25T05:26:16.130927+0000","last_fullsized":"2023-11-25T05:26:16.130927+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:38:50.120101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.706495+0000","last_change":"2023-11-25T05:05:51.087897+0000","last_active":"2023-11-25T05:26:12.706495+0000","last_peered":"2023-11-25T05:26:12.706495+0000","last_clean":"2023-11-25T05:26:12.706495+0000","last_became_active":"2023-11-25T05:05:50.968719+0000","last_became_peered":"2023-11-25T05:05:50.968719+0000","last_unstale":"2023-11-25T05:26:12.706495+0000","last_undegraded":"2023-11-25T05:26:12.706495+0000","last_fullsized":"2023-11-25T05:26:12.706495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:28:52.119629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":850,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.177301+0000","last_change":"2023-11-25T05:06:05.094337+0000","last_active":"2023-11-25T05:26:14.177301+0000","last_peered":"2023-11-25T05:26:14.177301+0000","last_clean":"2023-11-25T05:26:14.177301+0000","last_became_active":"2023-11-25T05:06:05.084111+0000","last_became_peered":"2023-11-25T05:06:05.084111+0000","last_unstale":"2023-11-25T05:26:14.177301+0000","last_undegraded":"2023-11-25T05:26:14.177301+0000","last_fullsized":"2023-11-25T05:26:14.177301+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T13:39:56.459513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":861,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.898941+0000","last_change":"2023-11-25T05:05:47.892163+0000","last_active":"2023-11-25T05:26:15.898941+0000","last_peered":"2023-11-25T05:26:15.898941+0000","last_clean":"2023-11-25T05:26:15.898941+0000","last_became_active":"2023-11-25T05:05:47.855679+0000","last_became_peered":"2023-11-25T05:05:47.855679+0000","last_unstale":"2023-11-25T05:26:15.898941+0000","last_undegraded":"2023-11-25T05:26:15.898941+0000","last_fullsized":"2023-11-25T05:26:15.898941+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:54:11.446291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":874,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.189467+0000","last_change":"2023-11-25T05:06:09.646267+0000","last_active":"2023-11-25T05:26:14.189467+0000","last_peered":"2023-11-25T05:26:14.189467+0000","last_clean":"2023-11-25T05:26:14.189467+0000","last_became_active":"2023-11-25T05:06:09.633454+0000","last_became_peered":"2023-11-25T05:06:09.633454+0000","last_unstale":"2023-11-25T05:26:14.189467+0000","last_undegraded":"2023-11-25T05:26:14.189467+0000","last_fullsized":"2023-11-25T05:26:14.189467+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:36:34.447428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":863,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.589752+0000","last_change":"2023-11-25T05:05:53.105351+0000","last_active":"2023-11-25T05:26:12.589752+0000","last_peered":"2023-11-25T05:26:12.589752+0000","last_clean":"2023-11-25T05:26:12.589752+0000","last_became_active":"2023-11-25T05:05:53.099253+0000","last_became_peered":"2023-11-25T05:05:53.099253+0000","last_unstale":"2023-11-25T05:26:12.589752+0000","last_undegraded":"2023-11-25T05:26:12.589752+0000","last_fullsized":"2023-11-25T05:26:12.589752+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:25:34.409386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":874,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.210137+0000","last_change":"2023-11-25T05:05:43.502596+0000","last_active":"2023-11-25T05:26:14.210137+0000","last_peered":"2023-11-25T05:26:14.210137+0000","last_clean":"2023-11-25T05:26:14.210137+0000","last_became_active":"2023-11-25T05:05:43.496305+0000","last_became_peered":"2023-11-25T05:05:43.496305+0000","last_unstale":"2023-11-25T05:26:14.210137+0000","last_undegraded":"2023-11-25T05:26:14.210137+0000","last_fullsized":"2023-11-25T05:26:14.210137+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:56:11.421935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":869,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.390854+0000","last_change":"2023-11-25T05:05:42.708461+0000","last_active":"2023-11-25T05:26:11.390854+0000","last_peered":"2023-11-25T05:26:11.390854+0000","last_clean":"2023-11-25T05:26:11.390854+0000","last_became_active":"2023-11-25T05:05:42.672388+0000","last_became_peered":"2023-11-25T05:05:42.672388+0000","last_unstale":"2023-11-25T05:26:11.390854+0000","last_undegraded":"2023-11-25T05:26:11.390854+0000","last_fullsized":"2023-11-25T05:26:11.390854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:34:44.252922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":870,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.132522+0000","last_change":"2023-11-25T05:05:46.966035+0000","last_active":"2023-11-25T05:26:16.132522+0000","last_peered":"2023-11-25T05:26:16.132522+0000","last_clean":"2023-11-25T05:26:16.132522+0000","last_became_active":"2023-11-25T05:05:46.959854+0000","last_became_peered":"2023-11-25T05:05:46.959854+0000","last_unstale":"2023-11-25T05:26:16.132522+0000","last_undegraded":"2023-11-25T05:26:16.132522+0000","last_fullsized":"2023-11-25T05:26:16.132522+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:00:30.622747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":869,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.400561+0000","last_change":"2023-11-25T05:05:47.759205+0000","last_active":"2023-11-25T05:26:11.400561+0000","last_peered":"2023-11-25T05:26:11.400561+0000","last_clean":"2023-11-25T05:26:11.400561+0000","last_became_active":"2023-11-25T05:05:47.753176+0000","last_became_peered":"2023-11-25T05:05:47.753176+0000","last_unstale":"2023-11-25T05:26:11.400561+0000","last_undegraded":"2023-11-25T05:26:11.400561+0000","last_fullsized":"2023-11-25T05:26:11.400561+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:04:21.487913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":870,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.427728+0000","last_change":"2023-11-25T05:05:45.604133+0000","last_active":"2023-11-25T05:26:11.427728+0000","last_peered":"2023-11-25T05:26:11.427728+0000","last_clean":"2023-11-25T05:26:11.427728+0000","last_became_active":"2023-11-25T05:05:45.597733+0000","last_became_peered":"2023-11-25T05:05:45.597733+0000","last_unstale":"2023-11-25T05:26:11.427728+0000","last_undegraded":"2023-11-25T05:26:11.427728+0000","last_fullsized":"2023-11-25T05:26:11.427728+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:20:05.883993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.604334+0000","last_change":"2023-11-25T05:05:47.799603+0000","last_active":"2023-11-25T05:26:13.604334+0000","last_peered":"2023-11-25T05:26:13.604334+0000","last_clean":"2023-11-25T05:26:13.604334+0000","last_became_active":"2023-11-25T05:05:47.792727+0000","last_became_peered":"2023-11-25T05:05:47.792727+0000","last_unstale":"2023-11-25T05:26:13.604334+0000","last_undegraded":"2023-11-25T05:26:13.604334+0000","last_fullsized":"2023-11-25T05:26:13.604334+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:50:46.409556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":866,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.923391+0000","last_change":"2023-11-25T05:04:33.582978+0000","last_active":"2023-11-25T05:26:15.923391+0000","last_peered":"2023-11-25T05:26:15.923391+0000","last_clean":"2023-11-25T05:26:15.923391+0000","last_became_active":"2023-11-25T05:04:33.577510+0000","last_became_peered":"2023-11-25T05:04:33.577510+0000","last_unstale":"2023-11-25T05:26:15.923391+0000","last_undegraded":"2023-11-25T05:26:15.923391+0000","last_fullsized":"2023-11-25T05:26:15.923391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:53:20.091826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":862,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.911955+0000","last_change":"2023-11-25T05:05:58.700776+0000","last_active":"2023-11-25T05:26:15.911955+0000","last_peered":"2023-11-25T05:26:15.911955+0000","last_clean":"2023-11-25T05:26:15.911955+0000","last_became_active":"2023-11-25T05:05:58.695110+0000","last_became_peered":"2023-11-25T05:05:58.695110+0000","last_unstale":"2023-11-25T05:26:15.911955+0000","last_undegraded":"2023-11-25T05:26:15.911955+0000","last_fullsized":"2023-11-25T05:26:15.911955+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T05:50:22.654582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":875,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.715982+0000","last_change":"2023-11-25T05:04:33.645680+0000","last_active":"2023-11-25T05:26:13.715982+0000","last_peered":"2023-11-25T05:26:13.715982+0000","last_clean":"2023-11-25T05:26:13.715982+0000","last_became_active":"2023-11-25T05:04:33.610776+0000","last_became_peered":"2023-11-25T05:04:33.610776+0000","last_unstale":"2023-11-25T05:26:13.715982+0000","last_undegraded":"2023-11-25T05:26:13.715982+0000","last_fullsized":"2023-11-25T05:26:13.715982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:02:08.013900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.680167+0000","last_change":"2023-11-25T05:06:12.667457+0000","last_active":"2023-11-25T05:26:12.680167+0000","last_peered":"2023-11-25T05:26:12.680167+0000","last_clean":"2023-11-25T05:26:12.680167+0000","last_became_active":"2023-11-25T05:06:12.543810+0000","last_became_peered":"2023-11-25T05:06:12.543810+0000","last_unstale":"2023-11-25T05:26:12.680167+0000","last_undegraded":"2023-11-25T05:26:12.680167+0000","last_fullsized":"2023-11-25T05:26:12.680167+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:13:23.439209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.910285+0000","last_change":"2023-11-25T05:04:33.350255+0000","last_active":"2023-11-25T05:26:15.910285+0000","last_peered":"2023-11-25T05:26:15.910285+0000","last_clean":"2023-11-25T05:26:15.910285+0000","last_became_active":"2023-11-25T05:04:33.337158+0000","last_became_peered":"2023-11-25T05:04:33.337158+0000","last_unstale":"2023-11-25T05:26:15.910285+0000","last_undegraded":"2023-11-25T05:26:15.910285+0000","last_fullsized":"2023-11-25T05:26:15.910285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:25:34.365788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.708724+0000","last_change":"2023-11-25T05:05:51.120038+0000","last_active":"2023-11-25T05:26:12.708724+0000","last_peered":"2023-11-25T05:26:12.708724+0000","last_clean":"2023-11-25T05:26:12.708724+0000","last_became_active":"2023-11-25T05:05:51.097150+0000","last_became_peered":"2023-11-25T05:05:51.097150+0000","last_unstale":"2023-11-25T05:26:12.708724+0000","last_undegraded":"2023-11-25T05:26:12.708724+0000","last_fullsized":"2023-11-25T05:26:12.708724+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:34:02.990987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":876,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.134189+0000","last_change":"2023-11-25T05:04:33.396056+0000","last_active":"2023-11-25T05:26:16.134189+0000","last_peered":"2023-11-25T05:26:16.134189+0000","last_clean":"2023-11-25T05:26:16.134189+0000","last_became_active":"2023-11-25T05:04:33.388849+0000","last_became_peered":"2023-11-25T05:04:33.388849+0000","last_unstale":"2023-11-25T05:26:16.134189+0000","last_undegraded":"2023-11-25T05:26:16.134189+0000","last_fullsized":"2023-11-25T05:26:16.134189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:11:08.763309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":871,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.120125+0000","last_change":"2023-11-25T05:05:55.046489+0000","last_active":"2023-11-25T05:26:16.120125+0000","last_peered":"2023-11-25T05:26:16.120125+0000","last_clean":"2023-11-25T05:26:16.120125+0000","last_became_active":"2023-11-25T05:05:55.038711+0000","last_became_peered":"2023-11-25T05:05:55.038711+0000","last_unstale":"2023-11-25T05:26:16.120125+0000","last_undegraded":"2023-11-25T05:26:16.120125+0000","last_fullsized":"2023-11-25T05:26:16.120125+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:43:58.709876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.902474+0000","last_change":"2023-11-25T05:04:33.381810+0000","last_active":"2023-11-25T05:26:15.902474+0000","last_peered":"2023-11-25T05:26:15.902474+0000","last_clean":"2023-11-25T05:26:15.902474+0000","last_became_active":"2023-11-25T05:04:33.376375+0000","last_became_peered":"2023-11-25T05:04:33.376375+0000","last_unstale":"2023-11-25T05:26:15.902474+0000","last_undegraded":"2023-11-25T05:26:15.902474+0000","last_fullsized":"2023-11-25T05:26:15.902474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:41:46.038487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":875,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.207787+0000","last_change":"2023-11-25T05:05:44.569896+0000","last_active":"2023-11-25T05:26:14.207787+0000","last_peered":"2023-11-25T05:26:14.207787+0000","last_clean":"2023-11-25T05:26:14.207787+0000","last_became_active":"2023-11-25T05:05:44.554983+0000","last_became_peered":"2023-11-25T05:05:44.554983+0000","last_unstale":"2023-11-25T05:26:14.207787+0000","last_undegraded":"2023-11-25T05:26:14.207787+0000","last_fullsized":"2023-11-25T05:26:14.207787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:59:24.121163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":871,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.564292+0000","last_change":"2023-11-25T05:04:33.238092+0000","last_active":"2023-11-25T05:26:13.564292+0000","last_peered":"2023-11-25T05:26:13.564292+0000","last_clean":"2023-11-25T05:26:13.564292+0000","last_became_active":"2023-11-25T05:04:33.196533+0000","last_became_peered":"2023-11-25T05:04:33.196533+0000","last_unstale":"2023-11-25T05:26:13.564292+0000","last_undegraded":"2023-11-25T05:26:13.564292+0000","last_fullsized":"2023-11-25T05:26:13.564292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:06:22.475692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":871,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.126082+0000","last_change":"2023-11-25T05:05:45.774913+0000","last_active":"2023-11-25T05:26:16.126082+0000","last_peered":"2023-11-25T05:26:16.126082+0000","last_clean":"2023-11-25T05:26:16.126082+0000","last_became_active":"2023-11-25T05:05:45.759824+0000","last_became_peered":"2023-11-25T05:05:45.759824+0000","last_unstale":"2023-11-25T05:26:16.126082+0000","last_undegraded":"2023-11-25T05:26:16.126082+0000","last_fullsized":"2023-11-25T05:26:16.126082+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:41:12.473852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.887432+0000","last_change":"2023-11-25T05:04:33.294027+0000","last_active":"2023-11-25T05:26:15.887432+0000","last_peered":"2023-11-25T05:26:15.887432+0000","last_clean":"2023-11-25T05:26:15.887432+0000","last_became_active":"2023-11-25T05:04:33.242068+0000","last_became_peered":"2023-11-25T05:04:33.242068+0000","last_unstale":"2023-11-25T05:26:15.887432+0000","last_undegraded":"2023-11-25T05:26:15.887432+0000","last_fullsized":"2023-11-25T05:26:15.887432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:37:54.756624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":871,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.106772+0000","last_change":"2023-11-25T05:05:45.087189+0000","last_active":"2023-11-25T05:26:16.106772+0000","last_peered":"2023-11-25T05:26:16.106772+0000","last_clean":"2023-11-25T05:26:16.106772+0000","last_became_active":"2023-11-25T05:05:45.057546+0000","last_became_peered":"2023-11-25T05:05:45.057546+0000","last_unstale":"2023-11-25T05:26:16.106772+0000","last_undegraded":"2023-11-25T05:26:16.106772+0000","last_fullsized":"2023-11-25T05:26:16.106772+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:02:04.146157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.907191+0000","last_change":"2023-11-25T05:04:33.784328+0000","last_active":"2023-11-25T05:26:15.907191+0000","last_peered":"2023-11-25T05:26:15.907191+0000","last_clean":"2023-11-25T05:26:15.907191+0000","last_became_active":"2023-11-25T05:04:33.778042+0000","last_became_peered":"2023-11-25T05:04:33.778042+0000","last_unstale":"2023-11-25T05:26:15.907191+0000","last_undegraded":"2023-11-25T05:26:15.907191+0000","last_fullsized":"2023-11-25T05:26:15.907191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:40:38.193703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":974,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.192116+0000","last_change":"2023-11-25T05:04:28.241367+0000","last_active":"2023-11-25T05:26:14.192116+0000","last_peered":"2023-11-25T05:26:14.192116+0000","last_clean":"2023-11-25T05:26:14.192116+0000","last_became_active":"2023-11-25T05:04:27.388156+0000","last_became_peered":"2023-11-25T05:04:27.388156+0000","last_unstale":"2023-11-25T05:26:14.192116+0000","last_undegraded":"2023-11-25T05:26:14.192116+0000","last_fullsized":"2023-11-25T05:26:14.192116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:22.310975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:22.310975+0000","last_clean_scrub_stamp":"2023-11-25T05:04:22.310975+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:50:37.122665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":870,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.404755+0000","last_change":"2023-11-25T05:05:47.996637+0000","last_active":"2023-11-25T05:26:11.404755+0000","last_peered":"2023-11-25T05:26:11.404755+0000","last_clean":"2023-11-25T05:26:11.404755+0000","last_became_active":"2023-11-25T05:05:47.973359+0000","last_became_peered":"2023-11-25T05:05:47.973359+0000","last_unstale":"2023-11-25T05:26:11.404755+0000","last_undegraded":"2023-11-25T05:26:11.404755+0000","last_fullsized":"2023-11-25T05:26:11.404755+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:05:25.523743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":852,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.171648+0000","last_change":"2023-11-25T05:05:54.064947+0000","last_active":"2023-11-25T05:26:14.171648+0000","last_peered":"2023-11-25T05:26:14.171648+0000","last_clean":"2023-11-25T05:26:14.171648+0000","last_became_active":"2023-11-25T05:05:54.045707+0000","last_became_peered":"2023-11-25T05:05:54.045707+0000","last_unstale":"2023-11-25T05:26:14.171648+0000","last_undegraded":"2023-11-25T05:26:14.171648+0000","last_fullsized":"2023-11-25T05:26:14.171648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T17:02:26.650184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":872,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.104094+0000","last_change":"2023-11-25T05:05:42.506167+0000","last_active":"2023-11-25T05:26:16.104094+0000","last_peered":"2023-11-25T05:26:16.104094+0000","last_clean":"2023-11-25T05:26:16.104094+0000","last_became_active":"2023-11-25T05:05:42.487757+0000","last_became_peered":"2023-11-25T05:05:42.487757+0000","last_unstale":"2023-11-25T05:26:16.104094+0000","last_undegraded":"2023-11-25T05:26:16.104094+0000","last_fullsized":"2023-11-25T05:26:16.104094+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:58:18.951826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"38'7","reported_seq":879,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.562585+0000","last_change":"2023-11-25T05:05:45.233028+0000","last_active":"2023-11-25T05:26:13.562585+0000","last_peered":"2023-11-25T05:26:13.562585+0000","last_clean":"2023-11-25T05:26:13.562585+0000","last_became_active":"2023-11-25T05:05:45.212870+0000","last_became_peered":"2023-11-25T05:05:45.212870+0000","last_unstale":"2023-11-25T05:26:13.562585+0000","last_undegraded":"2023-11-25T05:26:13.562585+0000","last_fullsized":"2023-11-25T05:26:13.562585+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:44:02.468772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.662501+0000","last_change":"2023-11-25T05:05:49.947608+0000","last_active":"2023-11-25T05:26:12.662501+0000","last_peered":"2023-11-25T05:26:12.662501+0000","last_clean":"2023-11-25T05:26:12.662501+0000","last_became_active":"2023-11-25T05:05:49.941593+0000","last_became_peered":"2023-11-25T05:05:49.941593+0000","last_unstale":"2023-11-25T05:26:12.662501+0000","last_undegraded":"2023-11-25T05:26:12.662501+0000","last_fullsized":"2023-11-25T05:26:12.662501+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:02:09.269520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.710445+0000","last_change":"2023-11-25T05:06:26.505899+0000","last_active":"2023-11-25T05:26:12.710445+0000","last_peered":"2023-11-25T05:26:12.710445+0000","last_clean":"2023-11-25T05:26:12.710445+0000","last_became_active":"2023-11-25T05:06:26.500023+0000","last_became_peered":"2023-11-25T05:06:26.500023+0000","last_unstale":"2023-11-25T05:26:12.710445+0000","last_undegraded":"2023-11-25T05:26:12.710445+0000","last_fullsized":"2023-11-25T05:26:12.710445+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T05:42:16.923983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.602721+0000","last_change":"2023-11-25T05:05:48.099018+0000","last_active":"2023-11-25T05:26:13.602721+0000","last_peered":"2023-11-25T05:26:13.602721+0000","last_clean":"2023-11-25T05:26:13.602721+0000","last_became_active":"2023-11-25T05:05:48.093083+0000","last_became_peered":"2023-11-25T05:05:48.093083+0000","last_unstale":"2023-11-25T05:26:13.602721+0000","last_undegraded":"2023-11-25T05:26:13.602721+0000","last_fullsized":"2023-11-25T05:26:13.602721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:36:31.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":870,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.139235+0000","last_change":"2023-11-25T05:05:55.013827+0000","last_active":"2023-11-25T05:26:16.139235+0000","last_peered":"2023-11-25T05:26:16.139235+0000","last_clean":"2023-11-25T05:26:16.139235+0000","last_became_active":"2023-11-25T05:05:55.006070+0000","last_became_peered":"2023-11-25T05:05:55.006070+0000","last_unstale":"2023-11-25T05:26:16.139235+0000","last_undegraded":"2023-11-25T05:26:16.139235+0000","last_fullsized":"2023-11-25T05:26:16.139235+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:42:44.573947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":870,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.123587+0000","last_change":"2023-11-25T05:05:42.334720+0000","last_active":"2023-11-25T05:26:16.123587+0000","last_peered":"2023-11-25T05:26:16.123587+0000","last_clean":"2023-11-25T05:26:16.123587+0000","last_became_active":"2023-11-25T05:05:42.327418+0000","last_became_peered":"2023-11-25T05:05:42.327418+0000","last_unstale":"2023-11-25T05:26:16.123587+0000","last_undegraded":"2023-11-25T05:26:16.123587+0000","last_fullsized":"2023-11-25T05:26:16.123587+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:45:59.399822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":869,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-11-25T05:26:13.679937+0000","last_change":"2023-11-25T05:05:45.341062+0000","last_active":"2023-11-25T05:26:13.679937+0000","last_peered":"2023-11-25T05:26:13.679937+0000","last_clean":"2023-11-25T05:26:13.679937+0000","last_became_active":"2023-11-25T05:05:45.334024+0000","last_became_peered":"2023-11-25T05:05:45.334024+0000","last_unstale":"2023-11-25T05:26:13.679937+0000","last_undegraded":"2023-11-25T05:26:13.679937+0000","last_fullsized":"2023-11-25T05:26:13.679937+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T05:06:56.822225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-11-25T05:26:12.582134+0000","last_change":"2023-11-25T05:06:27.132506+0000","last_active":"2023-11-25T05:26:12.582134+0000","last_peered":"2023-11-25T05:26:12.582134+0000","last_clean":"2023-11-25T05:26:12.582134+0000","last_became_active":"2023-11-25T05:06:27.122861+0000","last_became_peered":"2023-11-25T05:06:27.122861+0000","last_unstale":"2023-11-25T05:26:12.582134+0000","last_undegraded":"2023-11-25T05:26:12.582134+0000","last_fullsized":"2023-11-25T05:26:12.582134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:07:31.032076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":869,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-11-25T05:26:11.369980+0000","last_change":"2023-11-25T05:05:57.191530+0000","last_active":"2023-11-25T05:26:11.369980+0000","last_peered":"2023-11-25T05:26:11.369980+0000","last_clean":"2023-11-25T05:26:11.369980+0000","last_became_active":"2023-11-25T05:05:57.177615+0000","last_became_peered":"2023-11-25T05:05:57.177615+0000","last_unstale":"2023-11-25T05:26:11.369980+0000","last_undegraded":"2023-11-25T05:26:11.369980+0000","last_fullsized":"2023-11-25T05:26:11.369980+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:21:49.038926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":16,"seq":68719476999,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66640,"kb_used_data":13244,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305200,"statfs":{"total":96636764160,"available":96568524800,"internally_reserved":0,"allocated":13561856,"data_stored":8262098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":60,"apply_latency_ms":60,"commit_latency_ns":60000000,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":17.164000000000001,"5min":25.925000000000001,"15min":25.350999999999999},"min":{"1min":4.9020000000000001,"5min":4.9020000000000001,"15min":4.9020000000000001},"max":{"1min":100.782,"5min":164.57300000000001,"15min":203.28800000000001},"last":10.269},{"interface":"front","average":{"1min":15.237,"5min":26.946000000000002,"15min":26.018000000000001},"min":{"1min":5.7000000000000002,"5min":5.6859999999999999,"15min":5.4400000000000004},"max":{"1min":92.989000000000004,"5min":238.51300000000001,"15min":347.452},"last":17.202000000000002}]},{"osd":1,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":20.888000000000002,"5min":26.521999999999998,"15min":25.02},"min":{"1min":8.3249999999999993,"5min":7.4660000000000002,"15min":5.6589999999999998},"max":{"1min":86.962999999999994,"5min":287.27199999999999,"15min":287.27199999999999},"last":19.829999999999998},{"interface":"front","average":{"1min":17.350999999999999,"5min":27.222000000000001,"15min":23.896000000000001},"min":{"1min":5.282,"5min":5.282,"15min":5.282},"max":{"1min":57.453000000000003,"5min":286.57400000000001,"15min":286.57400000000001},"last":11.914}]},{"osd":3,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":23.404,"5min":20.637,"15min":20.605},"min":{"1min":6.3289999999999997,"5min":6.3289999999999997,"15min":5.9459999999999997},"max":{"1min":167.83799999999999,"5min":167.83799999999999,"15min":177.66},"last":9.5030000000000001},{"interface":"front","average":{"1min":30.977,"5min":28.187999999999999,"15min":31.155999999999999},"min":{"1min":11.647,"5min":8.1829999999999998,"15min":7.609},"max":{"1min":156.17699999999999,"5min":156.17699999999999,"15min":273.99900000000002},"last":15.653}]},{"osd":4,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":20.108000000000001,"5min":29.747,"15min":26.818000000000001},"min":{"1min":7.7789999999999999,"5min":6.9109999999999996,"15min":6.9109999999999996},"max":{"1min":71.611000000000004,"5min":275.66300000000001,"15min":349.90300000000002},"last":12.734999999999999},{"interface":"front","average":{"1min":17.427,"5min":26.164999999999999,"15min":22.431000000000001},"min":{"1min":7.1589999999999998,"5min":7.1589999999999998,"15min":6.6760000000000002},"max":{"1min":73.219999999999999,"5min":285.12700000000001,"15min":285.12700000000001},"last":7.8719999999999999}]},{"osd":5,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":22.722999999999999,"5min":24.5,"15min":29.382999999999999},"min":{"1min":11.710000000000001,"5min":9.7840000000000007,"15min":8.0619999999999994},"max":{"1min":89.492000000000004,"5min":138.34899999999999,"15min":353.63400000000001},"last":15.005000000000001},{"interface":"front","average":{"1min":26.366,"5min":30.738,"15min":34.677999999999997},"min":{"1min":14.381,"5min":13.541,"15min":11.539999999999999},"max":{"1min":96.620999999999995,"5min":146.41300000000001,"15min":354.56200000000001},"last":21.600999999999999}]},{"osd":6,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":31.605,"5min":30.536999999999999,"15min":31.172000000000001},"min":{"1min":16.321000000000002,"5min":11.292999999999999,"15min":9.7590000000000003},"max":{"1min":153.01400000000001,"5min":153.01400000000001,"15min":213.11000000000001},"last":20.943000000000001},{"interface":"front","average":{"1min":33.220999999999997,"5min":34.654000000000003,"15min":34.892000000000003},"min":{"1min":15.217000000000001,"5min":12.462999999999999,"15min":12.462999999999999},"max":{"1min":144.51400000000001,"5min":288.30000000000001,"15min":288.30000000000001},"last":22.247}]},{"osd":7,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":25.690999999999999,"5min":27.300000000000001,"15min":26.981000000000002},"min":{"1min":11.557,"5min":6.375,"15min":6.375},"max":{"1min":90.355999999999995,"5min":216.095,"15min":350.81599999999997},"last":17.853999999999999},{"interface":"front","average":{"1min":21.873000000000001,"5min":22.734000000000002,"15min":23.170000000000002},"min":{"1min":8.6400000000000006,"5min":7.5419999999999998,"15min":7.1669999999999998},"max":{"1min":96.100999999999999,"5min":141.59100000000001,"15min":210.55500000000001},"last":13.403}]}]},{"osd":1,"up_from":16,"seq":68719476999,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67056,"kb_used_data":13648,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94304784,"statfs":{"total":96636764160,"available":96568098816,"internally_reserved":0,"allocated":13975552,"data_stored":8666539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":23.388000000000002,"5min":20.16,"15min":21.632000000000001},"min":{"1min":6.165,"5min":5.9809999999999999,"15min":5.1600000000000001},"max":{"1min":163.62,"5min":221.589,"15min":349.75799999999998},"last":35.331000000000003},{"interface":"front","average":{"1min":32.435000000000002,"5min":29.16,"15min":26.466999999999999},"min":{"1min":9.4299999999999997,"5min":7.1529999999999996,"15min":6.5309999999999997},"max":{"1min":240.31200000000001,"5min":318.02600000000001,"15min":318.02600000000001},"last":28.692}]},{"osd":2,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":28.914000000000001,"5min":20.263000000000002,"15min":22.899999999999999},"min":{"1min":6.7889999999999997,"5min":5.1429999999999998,"15min":4.9619999999999997},"max":{"1min":148.28299999999999,"5min":161.10900000000001,"15min":353.286},"last":17.966000000000001},{"interface":"front","average":{"1min":34.442,"5min":21.100999999999999,"15min":19.414999999999999},"min":{"1min":7.7750000000000004,"5min":6.3280000000000003,"15min":5.7149999999999999},"max":{"1min":169.84100000000001,"5min":169.84100000000001,"15min":247.99199999999999},"last":30.224}]},{"osd":3,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":30.649000000000001,"5min":28.231999999999999,"15min":24.68},"min":{"1min":7.0339999999999998,"5min":6.665,"15min":6.5899999999999999},"max":{"1min":169.172,"5min":244.58199999999999,"15min":440.05599999999998},"last":33.240000000000002},{"interface":"front","average":{"1min":31.716000000000001,"5min":30.024000000000001,"15min":28.202999999999999},"min":{"1min":8.4369999999999994,"5min":8.4369999999999994,"15min":5.9889999999999999},"max":{"1min":168.58000000000001,"5min":242.29400000000001,"15min":383.39800000000002},"last":28.02}]},{"osd":4,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":20.422000000000001,"5min":17.603000000000002,"15min":21.190999999999999},"min":{"1min":8.9909999999999997,"5min":6.7830000000000004,"15min":6.5789999999999997},"max":{"1min":63.802999999999997,"5min":113.84,"15min":360.161},"last":34.058},{"interface":"front","average":{"1min":27.309999999999999,"5min":24.93,"15min":23.541},"min":{"1min":9.0510000000000002,"5min":9.0510000000000002,"15min":7.4409999999999998},"max":{"1min":114.33199999999999,"5min":226.68000000000001,"15min":364.39800000000002},"last":37.947000000000003}]},{"osd":5,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":39.484999999999999,"5min":28.780000000000001,"15min":29.396999999999998},"min":{"1min":9.6669999999999998,"5min":9.6669999999999998,"15min":7.4359999999999999},"max":{"1min":157.357,"5min":157.357,"15min":282.63900000000001},"last":99.200999999999993},{"interface":"front","average":{"1min":29.167999999999999,"5min":24.120999999999999,"15min":26.896000000000001},"min":{"1min":7.6630000000000003,"5min":7.2889999999999997,"15min":7.1529999999999996},"max":{"1min":105.211,"5min":163.33600000000001,"15min":382.01400000000001},"last":126.55}]},{"osd":6,"last update":"Sat Nov 25 05:25:21 2023","interfaces":[{"interface":"back","average":{"1min":18.364000000000001,"5min":24.588000000000001,"15min":47.482999999999997},"min":{"1min":10.593999999999999,"5min":7.5209999999999999,"15min":6.8230000000000004},"max":{"1min":64.594999999999999,"5min":187.62100000000001,"15min":3420.1689999999999},"last":32.476999999999997},{"interface":"front","average":{"1min":22.228999999999999,"5min":26.702000000000002,"15min":40.634999999999998},"min":{"1min":12.193,"5min":9.9260000000000002,"15min":9.423},"max":{"1min":70.814999999999998,"5min":131.99600000000001,"15min":1922.326},"last":34.700000000000003}]},{"osd":7,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":23.481999999999999,"5min":23.608000000000001,"15min":24.239999999999998},"min":{"1min":8.8559999999999999,"5min":7.0279999999999996,"15min":7.0279999999999996},"max":{"1min":116.39400000000001,"5min":219.57900000000001,"15min":272.03699999999998},"last":29.381},{"interface":"front","average":{"1min":30.045999999999999,"5min":26.843,"15min":27.969000000000001},"min":{"1min":11.65,"5min":10.239000000000001,"15min":8.3889999999999993},"max":{"1min":118.447,"5min":126.80200000000001,"15min":297.976},"last":35.962000000000003}]}]},{"osd":3,"up_from":16,"seq":68719477000,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84748,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94287092,"statfs":{"total":96636764160,"available":96549982208,"internally_reserved":0,"allocated":13598720,"data_stored":8282535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":21.867000000000001,"5min":16.334,"15min":21.956},"min":{"1min":5.556,"5min":4.3090000000000002,"15min":4.3090000000000002},"max":{"1min":100.45399999999999,"5min":124.46299999999999,"15min":394.84199999999998},"last":7.1159999999999997},{"interface":"front","average":{"1min":19.355,"5min":17.161000000000001,"15min":21.390999999999998},"min":{"1min":7.2969999999999997,"5min":4.9020000000000001,"15min":4.5890000000000004},"max":{"1min":84.197000000000003,"5min":119.611,"15min":379.51900000000001},"last":12.651}]},{"osd":1,"last update":"Sat Nov 25 05:26:14 2023","interfaces":[{"interface":"back","average":{"1min":21.181999999999999,"5min":20.585000000000001,"15min":28.129999999999999},"min":{"1min":7.6379999999999999,"5min":7.1779999999999999,"15min":5.4080000000000004},"max":{"1min":75.337999999999994,"5min":132.477,"15min":674.39599999999996},"last":11.920999999999999},{"interface":"front","average":{"1min":18.774000000000001,"5min":24.016999999999999,"15min":26.608000000000001},"min":{"1min":6.1829999999999998,"5min":6.1829999999999998,"15min":6.1829999999999998},"max":{"1min":62.155999999999999,"5min":253.03100000000001,"15min":399.98500000000001},"last":19.158999999999999}]},{"osd":2,"last update":"Sat Nov 25 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":18.132000000000001,"5min":16.486999999999998,"15min":34.378999999999998},"min":{"1min":7.798,"5min":7.2649999999999997,"15min":7.0620000000000003},"max":{"1min":83.954999999999998,"5min":83.954999999999998,"15min":673.12400000000002},"last":9.3879999999999999},{"interface":"front","average":{"1min":22.617000000000001,"5min":22.503,"15min":35.618000000000002},"min":{"1min":7.351,"5min":7.351,"15min":7.351},"max":{"1min":77.671999999999997,"5min":227.36199999999999,"15min":492.04899999999998},"last":14.470000000000001}]},{"osd":4,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":21.428999999999998,"5min":21.751999999999999,"15min":31.024000000000001},"min":{"1min":8.8360000000000003,"5min":8.8360000000000003,"15min":7.6890000000000001},"max":{"1min":81.578000000000003,"5min":141.38399999999999,"15min":676.21299999999997},"last":22.710000000000001},{"interface":"front","average":{"1min":19.236000000000001,"5min":20.013999999999999,"15min":23.495999999999999},"min":{"1min":7.6040000000000001,"5min":6.641,"15min":6.1559999999999997},"max":{"1min":77.034999999999997,"5min":313.18299999999999,"15min":313.18299999999999},"last":15.223000000000001}]},{"osd":5,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":26.818000000000001,"5min":23.085999999999999,"15min":32.587000000000003},"min":{"1min":10.710000000000001,"5min":9.6999999999999993,"15min":9.3360000000000003},"max":{"1min":80.661000000000001,"5min":194.51400000000001,"15min":675.37800000000004},"last":17.161000000000001},{"interface":"front","average":{"1min":27.657,"5min":21.199999999999999,"15min":28.454999999999998},"min":{"1min":11.975,"5min":7.5190000000000001,"15min":7.5190000000000001},"max":{"1min":103.021,"5min":131.768,"15min":314.40800000000002},"last":23.495000000000001}]},{"osd":6,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":23.952999999999999,"5min":20.739999999999998,"15min":24.882000000000001},"min":{"1min":8.5519999999999996,"5min":8.3030000000000008,"15min":7.0190000000000001},"max":{"1min":83.287999999999997,"5min":172.66,"15min":312.20699999999999},"last":13.022},{"interface":"front","average":{"1min":26.363,"5min":24.664000000000001,"15min":35.289999999999999},"min":{"1min":11.327,"5min":10.263999999999999,"15min":10.029},"max":{"1min":102.366,"5min":185.809,"15min":677.03499999999997},"last":21.332000000000001}]},{"osd":7,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":23.309000000000001,"5min":23.741,"15min":28.478999999999999},"min":{"1min":11.162000000000001,"5min":8.1609999999999996,"15min":7.6420000000000003},"max":{"1min":68.536000000000001,"5min":315.61500000000001,"15min":315.61500000000001},"last":15.903},{"interface":"front","average":{"1min":26.361999999999998,"5min":24.396000000000001,"15min":30.718},"min":{"1min":10.535,"5min":10.535,"15min":10.449999999999999},"max":{"1min":102.361,"5min":145.86099999999999,"15min":320.38400000000001},"last":25.646000000000001}]}]},{"osd":0,"up_from":16,"seq":68719477001,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66664,"kb_used_data":13256,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305176,"statfs":{"total":96636764160,"available":96568500224,"internally_reserved":0,"allocated":13574144,"data_stored":8268699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":12.904999999999999,"5min":15.397,"15min":17.724},"min":{"1min":4.6100000000000003,"5min":4.6100000000000003,"15min":4.6100000000000003},"max":{"1min":39.119,"5min":109.95399999999999,"15min":209.29900000000001},"last":23.010999999999999},{"interface":"front","average":{"1min":12.183999999999999,"5min":16.882000000000001,"15min":19.984999999999999},"min":{"1min":5.6219999999999999,"5min":4.5869999999999997,"15min":4.4420000000000002},"max":{"1min":56.362000000000002,"5min":119.73,"15min":401.36000000000001},"last":16.911000000000001}]},{"osd":2,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":17.683,"5min":19.295000000000002,"15min":22.402999999999999},"min":{"1min":6.6680000000000001,"5min":6.5380000000000003,"15min":6.1260000000000003},"max":{"1min":71.555999999999997,"5min":94.311999999999998,"15min":460.572},"last":35.844999999999999},{"interface":"front","average":{"1min":17.940000000000001,"5min":18.923999999999999,"15min":22.27},"min":{"1min":7.4980000000000002,"5min":6.5279999999999996,"15min":6.3090000000000002},"max":{"1min":73.290000000000006,"5min":89.322999999999993,"15min":405.94099999999997},"last":37.648000000000003}]},{"osd":3,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":21.096,"5min":23.585999999999999,"15min":24.260000000000002},"min":{"1min":7.2999999999999998,"5min":7.2999999999999998,"15min":6.8540000000000001},"max":{"1min":53.616999999999997,"5min":157.65299999999999,"15min":267.74599999999998},"last":29.195},{"interface":"front","average":{"1min":19.989000000000001,"5min":23.132000000000001,"15min":23.312000000000001},"min":{"1min":9.8480000000000008,"5min":7.1269999999999998,"15min":6.4080000000000004},"max":{"1min":55.122,"5min":128.14500000000001,"15min":266.81900000000002},"last":24.175999999999998}]},{"osd":4,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":21.091000000000001,"5min":20.103999999999999,"15min":20.606000000000002},"min":{"1min":9.4649999999999999,"5min":8.3940000000000001,"15min":6.2750000000000004},"max":{"1min":72.540999999999997,"5min":115.369,"15min":147.86099999999999},"last":11.446},{"interface":"front","average":{"1min":26.452000000000002,"5min":23.149999999999999,"15min":22.498999999999999},"min":{"1min":9.2189999999999994,"5min":6.8250000000000002,"15min":6.8250000000000002},"max":{"1min":110.93300000000001,"5min":127.291,"15min":148.96199999999999},"last":9.4199999999999999}]},{"osd":5,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.902999999999999,"5min":23.966999999999999,"15min":23.213999999999999},"min":{"1min":9.0109999999999992,"5min":6.9829999999999997,"15min":6.9829999999999997},"max":{"1min":69.957999999999998,"5min":184.80000000000001,"15min":184.80000000000001},"last":136},{"interface":"front","average":{"1min":20.024000000000001,"5min":23.408999999999999,"15min":25.248000000000001},"min":{"1min":10.836,"5min":7.3849999999999998,"15min":7.3849999999999998},"max":{"1min":58.210000000000001,"5min":131.583,"15min":395.65100000000001},"last":227.119}]},{"osd":6,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":21.329000000000001,"5min":23.550000000000001,"15min":24.617000000000001},"min":{"1min":9.2829999999999995,"5min":8.4809999999999999,"15min":8.4809999999999999},"max":{"1min":71.225999999999999,"5min":116.63800000000001,"15min":161.24299999999999},"last":18.061},{"interface":"front","average":{"1min":26.015999999999998,"5min":25.968,"15min":26.870000000000001},"min":{"1min":13.154999999999999,"5min":9.7759999999999998,"15min":9.7759999999999998},"max":{"1min":71.831999999999994,"5min":115.986,"15min":138.46799999999999},"last":15.566000000000001}]},{"osd":7,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.158000000000001,"5min":18.911000000000001,"15min":20.722999999999999},"min":{"1min":10.941000000000001,"5min":8.1340000000000003,"15min":7.1079999999999997},"max":{"1min":40.393999999999998,"5min":112.73099999999999,"15min":140.595},"last":21.469999999999999},{"interface":"front","average":{"1min":21.975999999999999,"5min":22.829999999999998,"15min":25.202999999999999},"min":{"1min":11.441000000000001,"5min":10.888999999999999,"15min":9.9860000000000007},"max":{"1min":57.088000000000001,"5min":103.19,"15min":339.298},"last":42.442999999999998}]}]},{"osd":6,"up_from":15,"seq":64424509704,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84768,"kb_used_data":13312,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94287072,"statfs":{"total":96636764160,"available":96549961728,"internally_reserved":0,"allocated":13631488,"data_stored":8340674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":16.198,"5min":17.219999999999999,"15min":19.709},"min":{"1min":4.7489999999999997,"5min":4.6740000000000004,"15min":4.4409999999999998},"max":{"1min":117.05500000000001,"5min":117.05500000000001,"15min":203.857},"last":25.263000000000002},{"interface":"front","average":{"1min":15.382999999999999,"5min":16.821000000000002,"15min":20.439},"min":{"1min":5.4820000000000002,"5min":5.1210000000000004,"15min":4.6289999999999996},"max":{"1min":117.795,"5min":236.78700000000001,"15min":381.43900000000002},"last":31.957999999999998}]},{"osd":1,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.943999999999999,"5min":22.879000000000001,"15min":26.055},"min":{"1min":8.0890000000000004,"5min":6.2839999999999998,"15min":6.165},"max":{"1min":143.61099999999999,"5min":229.53399999999999,"15min":310.423},"last":11.465999999999999},{"interface":"front","average":{"1min":20.885999999999999,"5min":20.399999999999999,"15min":26.382999999999999},"min":{"1min":6.3360000000000003,"5min":6.2030000000000003,"15min":5.1310000000000002},"max":{"1min":190.029,"5min":243.74700000000001,"15min":548.03599999999994},"last":5.9429999999999996}]},{"osd":2,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":28.033999999999999,"5min":22.832000000000001,"15min":27.013999999999999},"min":{"1min":7.9550000000000001,"5min":6.9690000000000003,"15min":6.9690000000000003},"max":{"1min":122.754,"5min":251.524,"15min":353.80399999999997},"last":90.819000000000003},{"interface":"front","average":{"1min":26.027000000000001,"5min":27.611999999999998,"15min":30.841999999999999},"min":{"1min":10.65,"5min":10.65,"15min":10.617000000000001},"max":{"1min":120.961,"5min":249.97399999999999,"15min":304.49900000000002},"last":90.048000000000002}]},{"osd":3,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":25.802,"5min":31.065999999999999,"15min":32.701999999999998},"min":{"1min":11.24,"5min":10.164,"15min":7.5419999999999998},"max":{"1min":185.28299999999999,"5min":259.89299999999997,"15min":295.82799999999997},"last":156.45099999999999},{"interface":"front","average":{"1min":22.247,"5min":26.661000000000001,"15min":26.797999999999998},"min":{"1min":7.7720000000000002,"5min":6.9199999999999999,"15min":6.0209999999999999},"max":{"1min":168.56100000000001,"5min":249.07400000000001,"15min":266.40199999999999},"last":181.51599999999999}]},{"osd":4,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":31.027999999999999,"5min":31.033000000000001,"15min":32.439999999999998},"min":{"1min":10.000999999999999,"5min":10.000999999999999,"15min":10.000999999999999},"max":{"1min":186.28,"5min":277.85000000000002,"15min":330.86900000000003},"last":12.423},{"interface":"front","average":{"1min":25.614999999999998,"5min":26.050000000000001,"15min":25.638000000000002},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":6.5919999999999996},"max":{"1min":192.25999999999999,"5min":288.13099999999997,"15min":288.13099999999997},"last":8.3740000000000006}]},{"osd":5,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.140000000000001,"5min":18.035,"15min":22.939},"min":{"1min":6.0039999999999996,"5min":5.5590000000000002,"15min":5.5590000000000002},"max":{"1min":121.31399999999999,"5min":237.46100000000001,"15min":281.13999999999999},"last":7.5819999999999999},{"interface":"front","average":{"1min":25.876000000000001,"5min":28.905999999999999,"15min":33.758000000000003},"min":{"1min":8.0069999999999997,"5min":8.0069999999999997,"15min":8.0069999999999997},"max":{"1min":133.44999999999999,"5min":269.77100000000002,"15min":311.09199999999998},"last":13.314}]},{"osd":7,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":29.199999999999999,"5min":31.532,"15min":34.887999999999998},"min":{"1min":13.699,"5min":12.319000000000001,"15min":9.2210000000000001},"max":{"1min":140.25200000000001,"5min":259.87700000000001,"15min":367.24900000000002},"last":14.119},{"interface":"front","average":{"1min":25.48,"5min":24.911999999999999,"15min":28.138999999999999},"min":{"1min":8.8529999999999998,"5min":8.3469999999999995,"15min":7.7199999999999998},"max":{"1min":161.37899999999999,"5min":269.14400000000001,"15min":325.54199999999997},"last":9.8339999999999996}]}]},{"osd":4,"up_from":15,"seq":64424509706,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70816,"kb_used_data":13260,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94301024,"statfs":{"total":96636764160,"available":96564248576,"internally_reserved":0,"allocated":13578240,"data_stored":8268704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":17.417999999999999,"5min":18.896000000000001,"15min":24.765999999999998},"min":{"1min":5.0439999999999996,"5min":4.3170000000000002,"15min":3.8279999999999998},"max":{"1min":103.83199999999999,"5min":150.73599999999999,"15min":361.67000000000002},"last":8.6720000000000006},{"interface":"front","average":{"1min":12.904,"5min":17.263999999999999,"15min":23.478999999999999},"min":{"1min":5.3449999999999998,"5min":4.5369999999999999,"15min":4.1289999999999996},"max":{"1min":55.915999999999997,"5min":216.86600000000001,"15min":270.81},"last":8.2230000000000008}]},{"osd":1,"last update":"Sat Nov 25 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":12.121,"5min":19.879999999999999,"15min":24.692},"min":{"1min":5.2149999999999999,"5min":5.0519999999999996,"15min":4.9749999999999996},"max":{"1min":58.972000000000001,"5min":205.36500000000001,"15min":445.61500000000001},"last":11.268000000000001},{"interface":"front","average":{"1min":15.630000000000001,"5min":20.486000000000001,"15min":24.024000000000001},"min":{"1min":10.656000000000001,"5min":6.5759999999999996,"15min":6.5759999999999996},"max":{"1min":30.831,"5min":155.90299999999999,"15min":274.81099999999998},"last":13.839}]},{"osd":2,"last update":"Sat Nov 25 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":15.132,"5min":20.442,"15min":28.692},"min":{"1min":9.8219999999999992,"5min":6.9320000000000004,"15min":6.9320000000000004},"max":{"1min":31.68,"5min":170.72200000000001,"15min":334.29399999999998},"last":12.66},{"interface":"front","average":{"1min":21.068000000000001,"5min":24.486000000000001,"15min":31.969999999999999},"min":{"1min":11.401,"5min":10.664,"15min":8.5899999999999999},"max":{"1min":74.477000000000004,"5min":165.952,"15min":348.54599999999999},"last":15.205}]},{"osd":3,"last update":"Sat Nov 25 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":14.75,"5min":24.774000000000001,"15min":37.012},"min":{"1min":8.0649999999999995,"5min":6.5810000000000004,"15min":6.133},"max":{"1min":40.886000000000003,"5min":206.09899999999999,"15min":588.27800000000002},"last":13.148999999999999},{"interface":"front","average":{"1min":16.039000000000001,"5min":25.753,"15min":28.155000000000001},"min":{"1min":6.9210000000000003,"5min":6.4740000000000002,"15min":6.4740000000000002},"max":{"1min":75.966999999999999,"5min":204.739,"15min":381.53100000000001},"last":6.54}]},{"osd":5,"last update":"Sat Nov 25 05:25:56 2023","interfaces":[{"interface":"back","average":{"1min":18.43,"5min":20.623999999999999,"15min":26.562000000000001},"min":{"1min":5.875,"5min":5.875,"15min":5.875},"max":{"1min":72.236999999999995,"5min":195.851,"15min":360.387},"last":16.718},{"interface":"front","average":{"1min":24.617000000000001,"5min":24.105,"15min":29.986999999999998},"min":{"1min":12.101000000000001,"5min":9.4469999999999992,"15min":9.4469999999999992},"max":{"1min":122.352,"5min":159.67099999999999,"15min":274.38200000000001},"last":18.823}]},{"osd":6,"last update":"Sat Nov 25 05:25:56 2023","interfaces":[{"interface":"back","average":{"1min":20.521000000000001,"5min":24.015000000000001,"15min":27.704999999999998},"min":{"1min":10.093,"5min":10.093,"15min":9.8409999999999993},"max":{"1min":48.161000000000001,"5min":169.44499999999999,"15min":181.43299999999999},"last":17.812999999999999},{"interface":"front","average":{"1min":24.329000000000001,"5min":22.428999999999998,"15min":23.622},"min":{"1min":15.603999999999999,"5min":7.1619999999999999,"15min":5.6390000000000002},"max":{"1min":60.040999999999997,"5min":212.25399999999999,"15min":219.26499999999999},"last":9.7159999999999993}]},{"osd":7,"last update":"Sat Nov 25 05:25:56 2023","interfaces":[{"interface":"back","average":{"1min":14.73,"5min":23.181000000000001,"15min":25.620999999999999},"min":{"1min":8.7100000000000009,"5min":7.0369999999999999,"15min":7.0369999999999999},"max":{"1min":27.728999999999999,"5min":202.363,"15min":215.512},"last":10.627000000000001},{"interface":"front","average":{"1min":14.417999999999999,"5min":20.943999999999999,"15min":24.698},"min":{"1min":6.0350000000000001,"5min":6.0350000000000001,"15min":6.0350000000000001},"max":{"1min":29.920999999999999,"5min":211.62200000000001,"15min":248.83199999999999},"last":14.505000000000001}]}]},{"osd":7,"up_from":13,"seq":55834575112,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66668,"kb_used_data":13272,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305172,"statfs":{"total":96636764160,"available":96568496128,"internally_reserved":0,"allocated":13590528,"data_stored":8295623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":11.608000000000001,"5min":13.819000000000001,"15min":17.303999999999998},"min":{"1min":4.6150000000000002,"5min":4.3579999999999997,"15min":4.3579999999999997},"max":{"1min":23.535,"5min":97.569000000000003,"15min":222.50800000000001},"last":28.766999999999999},{"interface":"front","average":{"1min":9.8520000000000003,"5min":15.375999999999999,"15min":18.193999999999999},"min":{"1min":4.7450000000000001,"5min":4.6600000000000001,"15min":4.5579999999999998},"max":{"1min":25.248000000000001,"5min":218.03999999999999,"15min":218.03999999999999},"last":34.869999999999997}]},{"osd":1,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.643000000000001,"5min":13.715999999999999,"15min":17.489000000000001},"min":{"1min":5.4859999999999998,"5min":5.4859999999999998,"15min":5.2619999999999996},"max":{"1min":79.078999999999994,"5min":79.078999999999994,"15min":165.99000000000001},"last":8.4079999999999995},{"interface":"front","average":{"1min":20.498000000000001,"5min":18.827000000000002,"15min":22.771999999999998},"min":{"1min":7.508,"5min":6.8140000000000001,"15min":6.7030000000000003},"max":{"1min":117.25,"5min":117.25,"15min":303.892},"last":10.375}]},{"osd":2,"last update":"Sat Nov 25 05:25:16 2023","interfaces":[{"interface":"back","average":{"1min":16.558,"5min":20.265000000000001,"15min":24.670999999999999},"min":{"1min":8.1530000000000005,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":26.408000000000001,"5min":186.08600000000001,"15min":214.37100000000001},"last":21.869},{"interface":"front","average":{"1min":13.426,"5min":19.419,"15min":23.486999999999998},"min":{"1min":7.8540000000000001,"5min":7.508,"15min":6.4279999999999999},"max":{"1min":23.759,"5min":192.584,"15min":337.41899999999998},"last":22.702000000000002}]},{"osd":3,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":16.559999999999999,"5min":22.437999999999999,"15min":23.844999999999999},"min":{"1min":12.122999999999999,"5min":8.4469999999999992,"15min":7.5199999999999996},"max":{"1min":28.463000000000001,"5min":152.47399999999999,"15min":238.941},"last":13.661},{"interface":"front","average":{"1min":18.123000000000001,"5min":21.559999999999999,"15min":26.684999999999999},"min":{"1min":12.221,"5min":6.6929999999999996,"15min":5.5949999999999998},"max":{"1min":29.536000000000001,"5min":150.208,"15min":1292.413},"last":17.611999999999998}]},{"osd":4,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.506,"5min":23.238,"15min":25.268000000000001},"min":{"1min":8.8109999999999999,"5min":8.8109999999999999,"15min":7.9340000000000002},"max":{"1min":21.449999999999999,"5min":120.414,"15min":328.31400000000002},"last":10.839},{"interface":"front","average":{"1min":27.879000000000001,"5min":21.405000000000001,"15min":23.079999999999998},"min":{"1min":10.739000000000001,"5min":7.1849999999999996,"15min":6.4320000000000004},"max":{"1min":192.684,"5min":192.684,"15min":302.97199999999998},"last":27.952000000000002}]},{"osd":5,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":17.332000000000001,"5min":22.279,"15min":24.291},"min":{"1min":9.1509999999999998,"5min":7.7919999999999998,"15min":7.7619999999999996},"max":{"1min":42.118000000000002,"5min":186.69499999999999,"15min":217.023},"last":21.094000000000001},{"interface":"front","average":{"1min":23.495000000000001,"5min":28.128,"15min":29.045999999999999},"min":{"1min":11.694000000000001,"5min":10.643000000000001,"15min":7.1130000000000004},"max":{"1min":83.798000000000002,"5min":210.22300000000001,"15min":318.11599999999999},"last":30.222000000000001}]},{"osd":6,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.75,"5min":20.798999999999999,"15min":22.933},"min":{"1min":7.069,"5min":6.2380000000000004,"15min":6.2380000000000004},"max":{"1min":100.828,"5min":188.15199999999999,"15min":282.28100000000001},"last":26.225999999999999},{"interface":"front","average":{"1min":18.027999999999999,"5min":23.329000000000001,"15min":25.382000000000001},"min":{"1min":8.0459999999999994,"5min":8.0459999999999994,"15min":8.0459999999999994},"max":{"1min":100.143,"5min":187.09899999999999,"15min":215.00700000000001},"last":47.826000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575112,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89288,"kb_used_data":13672,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94282552,"statfs":{"total":96636764160,"available":96545333248,"internally_reserved":0,"allocated":14000128,"data_stored":8680375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":78,"apply_latency_ms":78,"commit_latency_ns":78000000,"apply_latency_ns":78000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":17.623000000000001,"5min":14.541,"15min":18.605},"min":{"1min":4.5910000000000002,"5min":4.202,"15min":4.202},"max":{"1min":133.92599999999999,"5min":167.83600000000001,"15min":471.81},"last":29.609000000000002},{"interface":"front","average":{"1min":14.865,"5min":15.755000000000001,"15min":20.606999999999999},"min":{"1min":6.2699999999999996,"5min":5.4279999999999999,"15min":4.5529999999999999},"max":{"1min":46.905000000000001,"5min":197.834,"15min":317.45699999999999},"last":9.3529999999999998}]},{"osd":1,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":27.741,"5min":18.356000000000002,"15min":21.565000000000001},"min":{"1min":6.3470000000000004,"5min":5.4189999999999996,"15min":5.0709999999999997},"max":{"1min":140.28999999999999,"5min":140.28999999999999,"15min":263.589},"last":8.5440000000000005},{"interface":"front","average":{"1min":30.024000000000001,"5min":17.751000000000001,"15min":23.059999999999999},"min":{"1min":6.351,"5min":5.1459999999999999,"15min":5.1459999999999999},"max":{"1min":150.196,"5min":150.196,"15min":238.26499999999999},"last":10.954000000000001}]},{"osd":2,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":19.062999999999999,"5min":19.102,"15min":23.210000000000001},"min":{"1min":7.1890000000000001,"5min":6.2240000000000002,"15min":6.2240000000000002},"max":{"1min":82.295000000000002,"5min":182.477,"15min":416.83199999999999},"last":13.824999999999999},{"interface":"front","average":{"1min":28.707000000000001,"5min":25.138000000000002,"15min":25.091999999999999},"min":{"1min":12.006,"5min":9.4000000000000004,"15min":7.0880000000000001},"max":{"1min":179.749,"5min":190.517,"15min":207.28700000000001},"last":14.474}]},{"osd":3,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":31.044,"5min":24.114000000000001,"15min":29.751999999999999},"min":{"1min":10.327,"5min":10.097,"15min":6.6079999999999997},"max":{"1min":166.59700000000001,"5min":166.59700000000001,"15min":443.10300000000001},"last":12.470000000000001},{"interface":"front","average":{"1min":28.024999999999999,"5min":19.308,"15min":31.902000000000001},"min":{"1min":7.851,"5min":6.9800000000000004,"15min":6.2199999999999998},"max":{"1min":130.87299999999999,"5min":130.87299999999999,"15min":1666.6489999999999},"last":35.137999999999998}]},{"osd":4,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":30.039000000000001,"5min":25.219999999999999,"15min":25.515999999999998},"min":{"1min":10.957000000000001,"5min":10.657,"15min":9.0280000000000005},"max":{"1min":173.041,"5min":173.041,"15min":173.041},"last":11.816000000000001},{"interface":"front","average":{"1min":17.225999999999999,"5min":17.971,"15min":21.449000000000002},"min":{"1min":6.5330000000000004,"5min":6.5330000000000004,"15min":6.5330000000000004},"max":{"1min":68.790000000000006,"5min":123.01300000000001,"15min":156.92500000000001},"last":32.195999999999998}]},{"osd":6,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":26.797999999999998,"5min":26.663,"15min":31.244},"min":{"1min":13.439,"5min":11.666,"15min":9.9979999999999993},"max":{"1min":100.47799999999999,"5min":229.86199999999999,"15min":229.86199999999999},"last":15.119999999999999},{"interface":"front","average":{"1min":31.841999999999999,"5min":24.629000000000001,"15min":27.375},"min":{"1min":8.6280000000000001,"5min":8.1159999999999997,"15min":7.4770000000000003},"max":{"1min":165.46000000000001,"5min":234.31100000000001,"15min":234.31100000000001},"last":33.174999999999997}]},{"osd":7,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":25.68,"5min":20.594999999999999,"15min":26.533999999999999},"min":{"1min":7.2679999999999998,"5min":6.2370000000000001,"15min":6.2370000000000001},"max":{"1min":67.870999999999995,"5min":195.94399999999999,"15min":195.94399999999999},"last":13.129},{"interface":"front","average":{"1min":23.308,"5min":25.666,"15min":26.741},"min":{"1min":11.151,"5min":11.151,"15min":7.3289999999999997},"max":{"1min":57.329000000000001,"5min":185.74700000000001,"15min":224.99600000000001},"last":34.164000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-25T05:26:18.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-25T05:26:18.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.549+0000 7f2433c4b640 10 monclient: get_monmap_and_config 2023-11-25T05:26:18.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.549+0000 7f2433c4b640 10 monclient: build_initial_monmap 2023-11-25T05:26:18.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.549+0000 7f2433c4b640 10 monclient: monmap: 2023-11-25T05:26:18.551 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:18.551 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:18.551 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:18.550317+0000 2023-11-25T05:26:18.551 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:18.550317+0000 2023-11-25T05:26:18.551 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:18.551 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:18.552 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:18.552 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:18.552 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:18.552 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:18.552 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:18.552 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:18.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.549+0000 7f2433c4b640 1 Processor -- start 2023-11-25T05:26:18.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.550+0000 7f2433c4b640 1 -- start start 2023-11-25T05:26:18.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.550+0000 7f2433c4b640 10 monclient: init 2023-11-25T05:26:18.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.550+0000 7f2433c4b640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:18.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.550+0000 7f2433c4b640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-11-25T05:26:18.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.550+0000 7f2433c4b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f242c0a8950 0x7f242c13f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:18.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.550+0000 7f2433c4b640 10 monclient(hunting): picked mon.noname-a con 0x7f242c0a8950 addr v2:172.21.15.7:3300/0 2023-11-25T05:26:18.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.550+0000 7f2433c4b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f242c0a8d30 addr v1:172.21.15.90:6789/0 2023-11-25T05:26:18.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.550+0000 7f2433c4b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f242c145ec0 0x7f242c13f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:18.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f2433c4b640 10 monclient(hunting): picked mon.noname-b con 0x7f242c145ec0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:18.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f2433c4b640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242c145320 con 0x7f242c0a8d30 2023-11-25T05:26:18.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f2433c4b640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:18.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f2433c4b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f242c0b2260 con 0x7f242c0a8950 2023-11-25T05:26:18.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f2433c4b640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:18.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f2433c4b640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242c0aa5e0 con 0x7f242c145ec0 2023-11-25T05:26:18.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f2433c4b640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:18.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f2433c4b640 10 monclient(hunting): authenticate will time out at 2704.793884s 2023-11-25T05:26:18.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f24321e8640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f242c145ec0 0x7f242c13f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:18.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f24319e7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f242c0a8950 0x7f242c13f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:18.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f24311e6640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f242c0a8d30 0x7f242c1457b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:40496/0 (socket says 172.21.15.7:40496) 2023-11-25T05:26:18.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f24311e6640 1 -- 172.21.15.7:0/2153345414 learned_addr learned my addr 172.21.15.7:0/2153345414 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:26:18.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f24321e8640 10 monclient(hunting): get_auth_request con 0x7f242c145ec0 auth_method 0 2023-11-25T05:26:18.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f24321e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:18.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f24321e8640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:18.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.551+0000 7f24321e8640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:18.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.552+0000 7f24319e7640 10 monclient(hunting): get_auth_request con 0x7f242c0a8950 auth_method 0 2023-11-25T05:26:18.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.552+0000 7f24319e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:18.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.552+0000 7f24319e7640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:18.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.552+0000 7f24319e7640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:18.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.552+0000 7f24321e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:18.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.552+0000 7f24321e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:18.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.552+0000 7f24321e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:18.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.552+0000 7f24319e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:18.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.552+0000 7f24319e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:18.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.552+0000 7f24319e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:18.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.554+0000 7f24321e8640 10 monclient(hunting): handle_auth_done global_id 17372 payload 274 2023-11-25T05:26:18.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.554+0000 7f24321e8640 1 -- 172.21.15.7:0/2153345414 >> v2:172.21.15.7:3300/0 conn(0x7f242c0a8950 msgr2=0x7f242c13f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:18.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.554+0000 7f24321e8640 1 --2- 172.21.15.7:0/2153345414 >> v2:172.21.15.7:3300/0 conn(0x7f242c0a8950 0x7f242c13f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:18.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.554+0000 7f24321e8640 1 -- 172.21.15.7:0/2153345414 >> v1:172.21.15.90:6789/0 conn(0x7f242c0a8d30 legacy=0x7f242c1457b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:18.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.554+0000 7f24309e5640 1 -- 172.21.15.7:0/2153345414 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183480676 0 0) 0x7f242c145320 con 0x7f242c0a8d30 2023-11-25T05:26:18.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.554+0000 7f24321e8640 10 monclient: _finish_hunting 0 2023-11-25T05:26:18.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.554+0000 7f24321e8640 1 monclient: found mon.noname-b 2023-11-25T05:26:18.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.554+0000 7f24321e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:18.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.554+0000 7f24321e8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:26:18.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.554+0000 7f24321e8640 1 -- 172.21.15.7:0/2153345414 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c0aa8c0 con 0x7f242c145ec0 2023-11-25T05:26:18.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.554+0000 7f24309e5640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-11-25T05:26:18.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.556+0000 7f24321e8640 1 --2- 172.21.15.7:0/2153345414 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f242c145ec0 0x7f242c13f6d0 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f2424009d90 tx=0x7f242402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cff962ce3ba95f5d server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:18.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f24309e5640 1 -- 172.21.15.7:0/2153345414 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242402cc50 con 0x7f242c145ec0 2023-11-25T05:26:18.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f24309e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:18.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f24309e5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:18.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f24309e5640 10 monclient: dump: 2023-11-25T05:26:18.564 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:18.564 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:18.564 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:18.564 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:18.564 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:18.564 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:18.564 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:18.565 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:18.565 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:18.565 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:18.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f24309e5640 10 monclient: _finish_auth 0 2023-11-25T05:26:18.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f24309e5640 10 monclient: _check_auth_tickets 2023-11-25T05:26:18.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f24309e5640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:18.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f24309e5640 1 -- 172.21.15.7:0/2153345414 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242402cdb0 con 0x7f242c145ec0 2023-11-25T05:26:18.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f24309e5640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:18.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f24309e5640 1 -- 172.21.15.7:0/2153345414 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424034660 con 0x7f242c145ec0 2023-11-25T05:26:18.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f24309e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:18.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f24309e5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:26:18.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f24309e5640 10 monclient: dump: 2023-11-25T05:26:18.565 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:18.566 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:18.567 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f2433c4b640 5 monclient: authenticate success, global_id 17372 2023-11-25T05:26:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f2433c4b640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.559+0000 7f2433c4b640 4 set_mon_vals no callback set 2023-11-25T05:26:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.560+0000 7f2433c4b640 10 monclient: shutdown 2023-11-25T05:26:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.560+0000 7f2433c4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.560+0000 7f2433c4b640 1 -- 172.21.15.7:0/2153345414 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f242c145ec0 msgr2=0x7f242c13f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.560+0000 7f2433c4b640 1 --2- 172.21.15.7:0/2153345414 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f242c145ec0 0x7f242c13f6d0 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f2424009d90 tx=0x7f242402bea0 comp rx=0 tx=0).stop 2023-11-25T05:26:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.560+0000 7f2433c4b640 1 -- 172.21.15.7:0/2153345414 shutdown_connections 2023-11-25T05:26:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.560+0000 7f2433c4b640 1 --2- 172.21.15.7:0/2153345414 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f242c145ec0 0x7f242c13f6d0 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.560+0000 7f2433c4b640 1 --2- 172.21.15.7:0/2153345414 >> v2:172.21.15.7:3300/0 conn(0x7f242c0a8950 0x7f242c13f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.560+0000 7f2433c4b640 1 -- 172.21.15.7:0/2153345414 >> 172.21.15.7:0/2153345414 conn(0x7f242c061970 msgr2=0x7f242c0b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.560+0000 7f2433c4b640 1 -- 172.21.15.7:0/2153345414 shutdown_connections 2023-11-25T05:26:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.560+0000 7f2433c4b640 1 -- 172.21.15.7:0/2153345414 wait complete. 2023-11-25T05:26:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.561+0000 7f2433c4b640 10 monclient: build_initial_monmap 2023-11-25T05:26:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.561+0000 7f2433c4b640 10 monclient: monmap: 2023-11-25T05:26:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:18.569 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.561+0000 7f2433c4b640 1 librados: starting msgr at 2023-11-25T05:26:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.561+0000 7f2433c4b640 1 librados: starting objecter 2023-11-25T05:26:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.561+0000 7f2433c4b640 1 Processor -- start 2023-11-25T05:26:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.561+0000 7f2433c4b640 1 -- start start 2023-11-25T05:26:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.561+0000 7f2433c4b640 1 librados: setting wanted keys 2023-11-25T05:26:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.561+0000 7f2433c4b640 1 librados: calling monclient init 2023-11-25T05:26:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.561+0000 7f2433c4b640 10 monclient: init 2023-11-25T05:26:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-25T05:26:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f242c0a8950 0x7f242c137d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 10 monclient(hunting): picked mon.noname-b con 0x7f242c0a8950 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f242c0a8d30 0x7f242c1d7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 10 monclient(hunting): picked mon.noname-c con 0x7f242c0a8d30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f242c145ec0 0x7f242c1da460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 10 monclient(hunting): picked mon.noname-a con 0x7f242c145ec0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c1d70e0 con 0x7f242c145ec0 2023-11-25T05:26:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c1416c0 con 0x7f242c0a8950 2023-11-25T05:26:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242c1382b0 con 0x7f242c0a8d30 2023-11-25T05:26:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f2433c4b640 10 monclient(hunting): authenticate will time out at 2704.805202s 2023-11-25T05:26:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f24311e6640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f242c0a8d30 0x7f242c1d7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.562+0000 7f24319e7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f242c0a8950 0x7f242c137d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24321e8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f242c145ec0 0x7f242c1da460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24311e6640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f242c0a8d30 0x7f242c1d7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38216/0 (socket says 172.21.15.7:38216) 2023-11-25T05:26:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24311e6640 1 -- 172.21.15.7:0/3788574989 learned_addr learned my addr 172.21.15.7:0/3788574989 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24311e6640 10 monclient(hunting): get_auth_request con 0x7f242c0a8d30 auth_method 0 2023-11-25T05:26:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24311e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24311e6640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24311e6640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24319e7640 10 monclient(hunting): get_auth_request con 0x7f242c0a8950 auth_method 0 2023-11-25T05:26:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24319e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24319e7640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24319e7640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24321e8640 10 monclient(hunting): get_auth_request con 0x7f242c145ec0 auth_method 0 2023-11-25T05:26:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24321e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24321e8640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24321e8640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24311e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24311e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24311e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24319e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24319e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24319e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.563+0000 7f24321e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.564+0000 7f24321e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.564+0000 7f24321e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.566+0000 7f24319e7640 10 monclient(hunting): handle_auth_done global_id 21226 payload 824 2023-11-25T05:26:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.566+0000 7f24319e7640 1 -- 172.21.15.7:0/3788574989 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f242c0a8d30 msgr2=0x7f242c1d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.566+0000 7f24319e7640 1 --2- 172.21.15.7:0/3788574989 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f242c0a8d30 0x7f242c1d7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.566+0000 7f24319e7640 1 -- 172.21.15.7:0/3788574989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f242c145ec0 msgr2=0x7f242c1da460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.566+0000 7f24319e7640 1 --2- 172.21.15.7:0/3788574989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f242c145ec0 0x7f242c1da460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.566+0000 7f24319e7640 10 monclient: _finish_hunting 0 2023-11-25T05:26:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.566+0000 7f24319e7640 1 monclient: found mon.noname-b 2023-11-25T05:26:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.566+0000 7f24319e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.566+0000 7f24319e7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:26:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.566+0000 7f24319e7640 1 -- 172.21.15.7:0/3788574989 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2424009a70 con 0x7f242c0a8950 2023-11-25T05:26:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.566+0000 7f24311e6640 1 --2- 172.21.15.7:0/3788574989 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f242c0a8d30 0x7f242c1d7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-25T05:26:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.568+0000 7f24319e7640 1 --2- 172.21.15.7:0/3788574989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f242c0a8950 0x7f242c137d70 secure :-1 s=READY pgs=5148 cs=0 l=1 rev1=1 crypto rx=0x7f242800e850 tx=0x7f242800ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.568+0000 7f24227fc640 10 client.?.objecter ms_handle_connect 0x7f242c0a8950 2023-11-25T05:26:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.568+0000 7f24227fc640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f24227fc640 1 -- 172.21.15.7:0/3788574989 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24280095b0 con 0x7f242c0a8950 2023-11-25T05:26:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f24227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f24227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f24227fc640 10 monclient: dump: 2023-11-25T05:26:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:18.580 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f24227fc640 10 monclient: _finish_auth 0 2023-11-25T05:26:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f24227fc640 10 monclient: _check_auth_tickets 2023-11-25T05:26:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f24227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f24227fc640 1 -- 172.21.15.7:0/3788574989 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242800c940 con 0x7f242c0a8950 2023-11-25T05:26:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f24227fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f24227fc640 1 -- 172.21.15.7:0/3788574989 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242800ccd0 con 0x7f242c0a8950 2023-11-25T05:26:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f2433c4b640 5 monclient: authenticate success, global_id 21226 2023-11-25T05:26:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f2433c4b640 10 monclient: _renew_subs 2023-11-25T05:26:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f2433c4b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f2433c4b640 1 -- 172.21.15.7:0/3788574989 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242c13fc20 con 0x7f242c0a8950 2023-11-25T05:26:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f24227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f24227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f24227fc640 10 monclient: dump: 2023-11-25T05:26:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:18.582 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:18.582 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:18.582 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:18.582 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:18.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f2423fff640 4 set_mon_vals no callback set 2023-11-25T05:26:18.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f2433c4b640 10 client.21226.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:18.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f2433c4b640 10 monclient: _renew_subs 2023-11-25T05:26:18.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f2433c4b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:18.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f2433c4b640 1 -- 172.21.15.7:0/3788574989 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242c142470 con 0x7f242c0a8950 2023-11-25T05:26:18.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.572+0000 7f2433c4b640 1 librados: init done 2023-11-25T05:26:18.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.573+0000 7f2433c4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:18.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.573+0000 7f2433c4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:18.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.573+0000 7f2433c4b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:18.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.573+0000 7f2433c4b640 1 -- 172.21.15.7:0/3788574989 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242c0b7610 con 0x7f242c0a8950 2023-11-25T05:26:18.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.591+0000 7f24227fc640 1 -- 172.21.15.7:0/3788574989 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2428013470 con 0x7f242c0a8950 2023-11-25T05:26:18.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.591+0000 7f24227fc640 1 --2- 172.21.15.7:0/3788574989 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f241006c820 0x7f241006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:18.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.591+0000 7f24311e6640 1 --2- 172.21.15.7:0/3788574989 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f241006c820 0x7f241006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:18.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.591+0000 7f24311e6640 10 monclient: get_auth_request con 0x7f241006c820 auth_method 0 2023-11-25T05:26:18.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.592+0000 7f24311e6640 1 --2- 172.21.15.7:0/3788574989 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f241006c820 0x7f241006ece0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f242c13c0b0 tx=0x7f241c007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:18.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.592+0000 7f24227fc640 10 client.21226.objecter ms_handle_connect 0x7f241006c820 2023-11-25T05:26:18.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.593+0000 7f24227fc640 1 -- 172.21.15.7:0/3788574989 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 1..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f242800ce30 con 0x7f242c0a8950 2023-11-25T05:26:18.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.593+0000 7f24227fc640 10 client.21226.objecter ms_dispatch 0x7f242c000b90 osd_map(918..918 src has 1..918) v4 2023-11-25T05:26:18.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.593+0000 7f24227fc640 3 client.21226.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:18.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.593+0000 7f24227fc640 3 client.21226.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:18.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.593+0000 7f24227fc640 20 client.21226.objecter dump_active .. 0 homeless 2023-11-25T05:26:18.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.821+0000 7f24227fc640 1 -- 172.21.15.7:0/3788574989 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2428055020 con 0x7f242c0a8950 2023-11-25T05:26:18.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.821+0000 7f24227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:18.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.821+0000 7f24227fc640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:26:19.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.999+0000 7f2433c4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:26:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.999+0000 7f2433c4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:26:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.999+0000 7f2433c4b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:18.999+0000 7f2433c4b640 1 -- 172.21.15.7:0/3788574989 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f242c138470 con 0x7f242c0a8950 2023-11-25T05:26:19.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.040+0000 7f24227fc640 1 -- 172.21.15.7:0/3788574989 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v918) v1 ==== 74+0+17325 (secure 0 0 0) 0x7f2428021090 con 0x7f242c0a8950 2023-11-25T05:26:19.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.040+0000 7f24227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-25T05:26:19.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.040+0000 7f24227fc640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:26:19.043 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:26:19.043 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":918,"fsid":"59cf1eff-816d-4eda-b513-a2de7ea218f7","created":"2023-11-25T05:03:35.598582+0000","modified":"2023-11-25T05:26:15.997701+0000","last_up_change":"2023-11-25T05:04:24.817743+0000","last_in_change":"2023-11-25T05:03:41.565581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":102,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-25T05:04:21.643061+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-25T05:04:31.232489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"ceph_test_rados_api_asio","create_time":"2023-11-25T05:05:36.051633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5c87807e-7b82-4866-ba36-60b44a25f317","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":911,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6825","nonce":2271997805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6827","nonce":2271997805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6831","nonce":2271997805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":2271997805},{"type":"v1","addr":"172.21.15.7:6829","nonce":2271997805}]},"public_addr":"172.21.15.7:6825/2271997805","cluster_addr":"172.21.15.7:6827/2271997805","heartbeat_back_addr":"172.21.15.7:6831/2271997805","heartbeat_front_addr":"172.21.15.7:6829/2271997805","state":["exists","up"]},{"osd":1,"uuid":"0e252ab8-9524-4ccc-b08f-d95bbf05fbea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":911,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6801","nonce":3792760010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6803","nonce":3792760010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6807","nonce":3792760010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3792760010},{"type":"v1","addr":"172.21.15.7:6805","nonce":3792760010}]},"public_addr":"172.21.15.7:6801/3792760010","cluster_addr":"172.21.15.7:6803/3792760010","heartbeat_back_addr":"172.21.15.7:6807/3792760010","heartbeat_front_addr":"172.21.15.7:6805/3792760010","state":["exists","up"]},{"osd":2,"uuid":"ccda96b8-aa9e-4c99-8d84-bba19ca0483b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":911,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6809","nonce":1372486177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6811","nonce":1372486177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6815","nonce":1372486177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1372486177},{"type":"v1","addr":"172.21.15.7:6813","nonce":1372486177}]},"public_addr":"172.21.15.7:6809/1372486177","cluster_addr":"172.21.15.7:6811/1372486177","heartbeat_back_addr":"172.21.15.7:6815/1372486177","heartbeat_front_addr":"172.21.15.7:6813/1372486177","state":["exists","up"]},{"osd":3,"uuid":"e4b8700a-2e28-4e69-8577-465666b1b051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":911,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6817","nonce":59754388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6819","nonce":59754388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6823","nonce":59754388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":59754388},{"type":"v1","addr":"172.21.15.7:6821","nonce":59754388}]},"public_addr":"172.21.15.7:6817/59754388","cluster_addr":"172.21.15.7:6819/59754388","heartbeat_back_addr":"172.21.15.7:6823/59754388","heartbeat_front_addr":"172.21.15.7:6821/59754388","state":["exists","up"]},{"osd":4,"uuid":"36c88a76-0467-4e37-85b4-dfd5953855a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":907,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6823","nonce":1576115380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6829","nonce":1576115380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6833","nonce":1576115380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1576115380},{"type":"v1","addr":"172.21.15.90:6831","nonce":1576115380}]},"public_addr":"172.21.15.90:6823/1576115380","cluster_addr":"172.21.15.90:6829/1576115380","heartbeat_back_addr":"172.21.15.90:6833/1576115380","heartbeat_front_addr":"172.21.15.90:6831/1576115380","state":["exists","up"]},{"osd":5,"uuid":"ebf452a0-1b17-488a-bbea-2e953c932ac7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":911,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6811","nonce":132027349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6813","nonce":132027349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6817","nonce":132027349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":132027349},{"type":"v1","addr":"172.21.15.90:6815","nonce":132027349}]},"public_addr":"172.21.15.90:6811/132027349","cluster_addr":"172.21.15.90:6813/132027349","heartbeat_back_addr":"172.21.15.90:6817/132027349","heartbeat_front_addr":"172.21.15.90:6815/132027349","state":["exists","up"]},{"osd":6,"uuid":"5aaf35c3-5c06-45e5-b788-8d3207c47cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":911,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6819","nonce":3560048676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6821","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6822","nonce":3560048676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6828","nonce":3560048676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3560048676},{"type":"v1","addr":"172.21.15.90:6825","nonce":3560048676}]},"public_addr":"172.21.15.90:6819/3560048676","cluster_addr":"172.21.15.90:6822/3560048676","heartbeat_back_addr":"172.21.15.90:6828/3560048676","heartbeat_front_addr":"172.21.15.90:6825/3560048676","state":["exists","up"]},{"osd":7,"uuid":"d50dd23a-f864-4593-ac00-4d8afed21dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":911,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6803","nonce":815268397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6805","nonce":815268397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6809","nonce":815268397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":815268397},{"type":"v1","addr":"172.21.15.90:6807","nonce":815268397}]},"public_addr":"172.21.15.90:6803/815268397","cluster_addr":"172.21.15.90:6805/815268397","heartbeat_back_addr":"172.21.15.90:6809/815268397","heartbeat_front_addr":"172.21.15.90:6807/815268397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:21.918591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:21.817203+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:21.018791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:22.357339+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:19.776736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:19.168168+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:20.275331+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-25T05:04:19.564073+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":14,"snaps":[{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":64,"snaps":[{"begin":11,"length":2}]},{"pool":87,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":98,"snaps":[{"begin":2,"length":1},{"begin":6,"length":2}]},{"pool":104,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":107,"snaps":[{"begin":4,"length":3}]},{"pool":117,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":174,"snaps":[{"begin":12,"length":1}]},{"pool":183,"snaps":[{"begin":14,"length":1}]},{"pool":188,"snaps":[{"begin":17,"length":1}]},{"pool":191,"snaps":[{"begin":19,"length":1},{"begin":21,"length":1}]},{"pool":197,"snaps":[{"begin":25,"length":1}]},{"pool":240,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":4,"length":1}]},{"pool":244,"snaps":[{"begin":6,"length":1}]},{"pool":248,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-25T05:26:19.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.042+0000 7f2433c4b640 10 librados: watch_flush enter 2023-11-25T05:26:19.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.042+0000 7f2433c4b640 10 librados: watch_flush exit 2023-11-25T05:26:19.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.042+0000 7f2433c4b640 20 client.21226.objecter shutdown clearing up homeless session... 2023-11-25T05:26:19.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.042+0000 7f2433c4b640 10 client.21226.objecter successfully canceled tick 2023-11-25T05:26:19.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.042+0000 7f2433c4b640 1 -- 172.21.15.7:0/3788574989 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f241006c820 msgr2=0x7f241006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:19.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.042+0000 7f2433c4b640 1 --2- 172.21.15.7:0/3788574989 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f241006c820 0x7f241006ece0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f242c13c0b0 tx=0x7f241c007de0 comp rx=0 tx=0).stop 2023-11-25T05:26:19.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.042+0000 7f2433c4b640 10 monclient: shutdown 2023-11-25T05:26:19.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.042+0000 7f2433c4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:19.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.042+0000 7f2433c4b640 1 -- 172.21.15.7:0/3788574989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f242c0a8950 msgr2=0x7f242c137d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:19.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.042+0000 7f2433c4b640 1 --2- 172.21.15.7:0/3788574989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f242c0a8950 0x7f242c137d70 secure :-1 s=READY pgs=5148 cs=0 l=1 rev1=1 crypto rx=0x7f242800e850 tx=0x7f242800ed10 comp rx=0 tx=0).stop 2023-11-25T05:26:19.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.043+0000 7f2433c4b640 1 -- 172.21.15.7:0/3788574989 shutdown_connections 2023-11-25T05:26:19.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.043+0000 7f2433c4b640 1 --2- 172.21.15.7:0/3788574989 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f241006c820 0x7f241006ece0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:19.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.043+0000 7f2433c4b640 1 --2- 172.21.15.7:0/3788574989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f242c145ec0 0x7f242c1da460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:19.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.043+0000 7f2433c4b640 1 --2- 172.21.15.7:0/3788574989 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f242c0a8d30 0x7f242c1d7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:19.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.043+0000 7f2433c4b640 1 --2- 172.21.15.7:0/3788574989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f242c0a8950 0x7f242c137d70 unknown :-1 s=CLOSED pgs=5148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:19.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.043+0000 7f2433c4b640 1 -- 172.21.15.7:0/3788574989 >> 172.21.15.7:0/3788574989 conn(0x7f242c061970 msgr2=0x7f242c0b2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:19.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.043+0000 7f2433c4b640 1 -- 172.21.15.7:0/3788574989 shutdown_connections 2023-11-25T05:26:19.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.043+0000 7f2433c4b640 1 -- 172.21.15.7:0/3788574989 wait complete. 2023-11-25T05:26:19.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:19.043+0000 7f2433c4b640 1 librados: shutdown 2023-11-25T05:26:20.057 INFO:tasks.ceph:Scrubbing osd.0 2023-11-25T05:26:20.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-11-25T05:26:20.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.170+0000 7f6b08d5a640 10 monclient: get_monmap_and_config 2023-11-25T05:26:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.170+0000 7f6b08d5a640 10 monclient: build_initial_monmap 2023-11-25T05:26:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.170+0000 7f6b08d5a640 10 monclient: monmap: 2023-11-25T05:26:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:20.171453+0000 2023-11-25T05:26:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:20.171453+0000 2023-11-25T05:26:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:20.173 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 1 Processor -- start 2023-11-25T05:26:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 1 -- start start 2023-11-25T05:26:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 10 monclient: init 2023-11-25T05:26:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-11-25T05:26:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b040af360 0x7f6b04143040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 10 monclient(hunting): picked mon.noname-a con 0x7f6b040af360 addr v2:172.21.15.7:3300/0 2023-11-25T05:26:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b04143580 0x7f6b0413cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 10 monclient(hunting): picked mon.noname-b con 0x7f6b04143580 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6b040720f0 0x7f6b0413d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 10 monclient(hunting): picked mon.noname-c con 0x7f6b040720f0 addr v2:172.21.15.90:3300/0 2023-11-25T05:26:20.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:20.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b040af740 con 0x7f6b040af360 2023-11-25T05:26:20.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:20.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b04071c60 con 0x7f6b040720f0 2023-11-25T05:26:20.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:20.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040a8c70 con 0x7f6b04143580 2023-11-25T05:26:20.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:20.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.171+0000 7f6b08d5a640 10 monclient(hunting): authenticate will time out at 2706.414740s 2023-11-25T05:26:20.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.172+0000 7f6afbfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b04143580 0x7f6b0413cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:20.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.172+0000 7f6b0259c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b040af360 0x7f6b04143040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:20.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.172+0000 7f6afbfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b04143580 0x7f6b0413cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38224/0 (socket says 172.21.15.7:38224) 2023-11-25T05:26:20.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.172+0000 7f6afbfff640 1 -- 172.21.15.7:0/890919009 learned_addr learned my addr 172.21.15.7:0/890919009 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:20.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.172+0000 7f6afbfff640 10 monclient(hunting): get_auth_request con 0x7f6b04143580 auth_method 0 2023-11-25T05:26:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.172+0000 7f6afbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.172+0000 7f6afbfff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.172+0000 7f6afbfff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.172+0000 7f6b02d9d640 1 --2- 172.21.15.7:0/890919009 >> v2:172.21.15.90:3300/0 conn(0x7f6b040720f0 0x7f6b0413d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.172+0000 7f6b0259c640 10 monclient(hunting): get_auth_request con 0x7f6b040af360 auth_method 0 2023-11-25T05:26:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.172+0000 7f6b0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.172+0000 7f6b0259c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.172+0000 7f6b0259c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.173+0000 7f6b02d9d640 10 monclient(hunting): get_auth_request con 0x7f6b040720f0 auth_method 0 2023-11-25T05:26:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.173+0000 7f6b02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.173+0000 7f6b02d9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.173+0000 7f6b02d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:20.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.173+0000 7f6afbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:20.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.173+0000 7f6afbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:20.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.173+0000 7f6afbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:20.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.173+0000 7f6b0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:20.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.173+0000 7f6b0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:20.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.173+0000 7f6b0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:20.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.173+0000 7f6b02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:20.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.173+0000 7f6b02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:20.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.173+0000 7f6b02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:20.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.174+0000 7f6afbfff640 10 monclient(hunting): handle_auth_done global_id 17378 payload 274 2023-11-25T05:26:20.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.174+0000 7f6afbfff640 1 -- 172.21.15.7:0/890919009 >> v2:172.21.15.90:3300/0 conn(0x7f6b040720f0 msgr2=0x7f6b0413d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:20.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.174+0000 7f6afbfff640 1 --2- 172.21.15.7:0/890919009 >> v2:172.21.15.90:3300/0 conn(0x7f6b040720f0 0x7f6b0413d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:20.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.175+0000 7f6afbfff640 1 -- 172.21.15.7:0/890919009 >> v2:172.21.15.7:3300/0 conn(0x7f6b040af360 msgr2=0x7f6b04143040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:20.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.175+0000 7f6afbfff640 1 --2- 172.21.15.7:0/890919009 >> v2:172.21.15.7:3300/0 conn(0x7f6b040af360 0x7f6b04143040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:20.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.175+0000 7f6afbfff640 10 monclient: _finish_hunting 0 2023-11-25T05:26:20.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.175+0000 7f6afbfff640 1 monclient: found mon.noname-b 2023-11-25T05:26:20.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.175+0000 7f6afbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:20.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.175+0000 7f6afbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:26:20.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.175+0000 7f6afbfff640 1 -- 172.21.15.7:0/890919009 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0413d6d0 con 0x7f6b04143580 2023-11-25T05:26:20.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.176+0000 7f6afbfff640 1 --2- 172.21.15.7:0/890919009 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b04143580 0x7f6b0413cc50 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7f6b040b0dd0 tx=0x7f6aec02b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a5d0daddce6ad0d server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:20.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b01d9b640 1 -- 172.21.15.7:0/890919009 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec031870 con 0x7f6b04143580 2023-11-25T05:26:20.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b01d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:20.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b01d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:20.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b01d9b640 10 monclient: dump: 2023-11-25T05:26:20.183 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:20.183 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:20.183 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.183 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.183 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:20.184 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:20.184 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:20.184 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:20.184 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:20.184 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:20.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b01d9b640 10 monclient: _finish_auth 0 2023-11-25T05:26:20.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b01d9b640 10 monclient: _check_auth_tickets 2023-11-25T05:26:20.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b01d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:20.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b01d9b640 1 -- 172.21.15.7:0/890919009 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aec0319d0 con 0x7f6b04143580 2023-11-25T05:26:20.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b08d5a640 5 monclient: authenticate success, global_id 17378 2023-11-25T05:26:20.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b08d5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:26:20.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b01d9b640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:20.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b08d5a640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:20.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b01d9b640 1 -- 172.21.15.7:0/890919009 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec031d60 con 0x7f6b04143580 2023-11-25T05:26:20.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b01d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:20.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b08d5a640 4 set_mon_vals no callback set 2023-11-25T05:26:20.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b01d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:26:20.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.180+0000 7f6b01d9b640 10 monclient: dump: 2023-11-25T05:26:20.185 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:20.185 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:20.186 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.186 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.186 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:20.186 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:20.186 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:20.186 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:20.186 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:20.186 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:20.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.181+0000 7f6b08d5a640 10 monclient: shutdown 2023-11-25T05:26:20.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.181+0000 7f6b08d5a640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:20.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.181+0000 7f6b08d5a640 1 -- 172.21.15.7:0/890919009 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b04143580 msgr2=0x7f6b0413cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:20.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.181+0000 7f6b08d5a640 1 --2- 172.21.15.7:0/890919009 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b04143580 0x7f6b0413cc50 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7f6b040b0dd0 tx=0x7f6aec02b910 comp rx=0 tx=0).stop 2023-11-25T05:26:20.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.181+0000 7f6b08d5a640 1 -- 172.21.15.7:0/890919009 shutdown_connections 2023-11-25T05:26:20.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.181+0000 7f6b08d5a640 1 --2- 172.21.15.7:0/890919009 >> v2:172.21.15.90:3300/0 conn(0x7f6b040720f0 0x7f6b0413d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:20.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.181+0000 7f6b08d5a640 1 --2- 172.21.15.7:0/890919009 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b04143580 0x7f6b0413cc50 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:20.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.181+0000 7f6b08d5a640 1 --2- 172.21.15.7:0/890919009 >> v2:172.21.15.7:3300/0 conn(0x7f6b040af360 0x7f6b04143040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:20.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.181+0000 7f6b08d5a640 1 -- 172.21.15.7:0/890919009 >> 172.21.15.7:0/890919009 conn(0x7f6b040b01e0 msgr2=0x7f6b040b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:20.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.181+0000 7f6b08d5a640 1 -- 172.21.15.7:0/890919009 shutdown_connections 2023-11-25T05:26:20.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.181+0000 7f6b08d5a640 1 -- 172.21.15.7:0/890919009 wait complete. 2023-11-25T05:26:20.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.182+0000 7f6b08d5a640 10 monclient: build_initial_monmap 2023-11-25T05:26:20.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.182+0000 7f6b08d5a640 10 monclient: monmap: 2023-11-25T05:26:20.191 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:20.191 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:20.191 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:20.191 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:20.191 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:20.191 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:20.191 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:20.191 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:20.191 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:20.191 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:20.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.182+0000 7f6b08d5a640 1 librados: starting msgr at 2023-11-25T05:26:20.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.182+0000 7f6b08d5a640 1 librados: starting objecter 2023-11-25T05:26:20.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.182+0000 7f6b08d5a640 1 Processor -- start 2023-11-25T05:26:20.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.182+0000 7f6b08d5a640 1 -- start start 2023-11-25T05:26:20.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.182+0000 7f6b08d5a640 1 librados: setting wanted keys 2023-11-25T05:26:20.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.182+0000 7f6b08d5a640 1 librados: calling monclient init 2023-11-25T05:26:20.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.182+0000 7f6b08d5a640 10 monclient: init 2023-11-25T05:26:20.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:20.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:26:20.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b040720f0 0x7f6b041d82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:20.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 10 monclient(hunting): picked mon.noname-a con 0x7f6b040720f0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:20.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b040af360 0x7f6b040acf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:20.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 10 monclient(hunting): picked mon.noname-c con 0x7f6b040af360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:20.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b04143580 0x7f6b040aa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:20.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 10 monclient(hunting): picked mon.noname-b con 0x7f6b04143580 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:20.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:20.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040ac7f0 con 0x7f6b040720f0 2023-11-25T05:26:20.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:20.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b041d8800 con 0x7f6b04143580 2023-11-25T05:26:20.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:20.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040aca20 con 0x7f6b040af360 2023-11-25T05:26:20.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:20.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6b08d5a640 10 monclient(hunting): authenticate will time out at 2706.426273s 2023-11-25T05:26:20.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.183+0000 7f6afbfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b040af360 0x7f6b040acf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:20.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b0259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b040720f0 0x7f6b041d82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:20.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b02d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b04143580 0x7f6b040aa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:20.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6afbfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b040af360 0x7f6b040acf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38226/0 (socket says 172.21.15.7:38226) 2023-11-25T05:26:20.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6afbfff640 1 -- 172.21.15.7:0/2816409301 learned_addr learned my addr 172.21.15.7:0/2816409301 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:20.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6afbfff640 10 monclient(hunting): get_auth_request con 0x7f6b040af360 auth_method 0 2023-11-25T05:26:20.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6afbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:20.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6afbfff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:20.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6afbfff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:20.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b02d9d640 10 monclient(hunting): get_auth_request con 0x7f6b04143580 auth_method 0 2023-11-25T05:26:20.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:20.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b02d9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:20.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b02d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:20.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b0259c640 10 monclient(hunting): get_auth_request con 0x7f6b040720f0 auth_method 0 2023-11-25T05:26:20.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:20.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b0259c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:20.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b0259c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:20.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6afbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:20.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6afbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:20.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6afbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:20.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:20.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:20.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:20.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:20.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:20.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.184+0000 7f6b0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:20.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.187+0000 7f6b02d9d640 10 monclient(hunting): handle_auth_done global_id 21232 payload 824 2023-11-25T05:26:20.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.187+0000 7f6b02d9d640 1 -- 172.21.15.7:0/2816409301 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b040af360 msgr2=0x7f6b040acf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:20.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.187+0000 7f6b02d9d640 1 --2- 172.21.15.7:0/2816409301 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6b040af360 0x7f6b040acf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:20.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.187+0000 7f6b02d9d640 1 -- 172.21.15.7:0/2816409301 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b040720f0 msgr2=0x7f6b041d82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:20.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.187+0000 7f6b02d9d640 1 --2- 172.21.15.7:0/2816409301 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b040720f0 0x7f6b041d82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:20.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.187+0000 7f6b02d9d640 10 monclient: _finish_hunting 0 2023-11-25T05:26:20.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.187+0000 7f6b02d9d640 1 monclient: found mon.noname-b 2023-11-25T05:26:20.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.187+0000 7f6b02d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:20.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.187+0000 7f6b02d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:26:20.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.187+0000 7f6b02d9d640 1 -- 172.21.15.7:0/2816409301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec0094d0 con 0x7f6b04143580 2023-11-25T05:26:20.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.189+0000 7f6b02d9d640 1 --2- 172.21.15.7:0/2816409301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b04143580 0x7f6b040aa1e0 secure :-1 s=READY pgs=5149 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00efc0 tx=0x7f6afc00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:20.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.189+0000 7f6afaffd640 10 client.?.objecter ms_handle_connect 0x7f6b04143580 2023-11-25T05:26:20.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.189+0000 7f6afaffd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:20.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6afaffd640 1 -- 172.21.15.7:0/2816409301 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc009c00 con 0x7f6b04143580 2023-11-25T05:26:20.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6afaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:20.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6afaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:20.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6afaffd640 10 monclient: dump: 2023-11-25T05:26:20.199 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:20.199 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:20.199 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.199 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.199 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:20.199 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:20.199 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:20.199 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:20.199 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:20.200 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6afaffd640 10 monclient: _finish_auth 0 2023-11-25T05:26:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6afaffd640 10 monclient: _check_auth_tickets 2023-11-25T05:26:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6afaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6afaffd640 1 -- 172.21.15.7:0/2816409301 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6afc00f040 con 0x7f6b04143580 2023-11-25T05:26:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6afaffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6afaffd640 1 -- 172.21.15.7:0/2816409301 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc0146c0 con 0x7f6b04143580 2023-11-25T05:26:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6b08d5a640 5 monclient: authenticate success, global_id 21232 2023-11-25T05:26:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6b08d5a640 10 monclient: _renew_subs 2023-11-25T05:26:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6b08d5a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6b08d5a640 1 -- 172.21.15.7:0/2816409301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b040ad480 con 0x7f6b04143580 2023-11-25T05:26:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6afaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:20.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6b0159a640 4 set_mon_vals no callback set 2023-11-25T05:26:20.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6afaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:20.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6afaffd640 10 monclient: dump: 2023-11-25T05:26:20.201 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:20.201 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:20.201 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.201 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.201 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:20.201 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:20.201 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:20.201 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:20.202 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:20.202 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:20.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6b08d5a640 10 client.21232.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:20.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6b08d5a640 10 monclient: _renew_subs 2023-11-25T05:26:20.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6b08d5a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:20.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6b08d5a640 1 -- 172.21.15.7:0/2816409301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b040aac20 con 0x7f6b04143580 2023-11-25T05:26:20.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.193+0000 7f6b08d5a640 1 librados: init done 2023-11-25T05:26:20.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.194+0000 7f6b08d5a640 10 client.21232.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:20.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.194+0000 7f6b08d5a640 15 client.21232.objecter _session_command_op_assign -1 1 2023-11-25T05:26:20.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.194+0000 7f6b08d5a640 20 client.21232.objecter _get_session osd=-1 returning homeless 2023-11-25T05:26:20.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.194+0000 7f6b08d5a640 10 client.21232.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:20.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.194+0000 7f6b08d5a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:20.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.194+0000 7f6b08d5a640 1 -- 172.21.15.7:0/2816409301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6b040acc40 con 0x7f6b04143580 2023-11-25T05:26:20.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.212+0000 7f6afaffd640 1 -- 172.21.15.7:0/2816409301 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6afc014820 con 0x7f6b04143580 2023-11-25T05:26:20.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.213+0000 7f6afaffd640 1 --2- 172.21.15.7:0/2816409301 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6ae406c7d0 0x7f6ae406ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:20.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.213+0000 7f6b0259c640 1 --2- 172.21.15.7:0/2816409301 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6ae406c7d0 0x7f6ae406ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:20.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.213+0000 7f6b0259c640 10 monclient: get_auth_request con 0x7f6ae406c7d0 auth_method 0 2023-11-25T05:26:20.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.213+0000 7f6afaffd640 10 client.21232.objecter ms_handle_connect 0x7f6ae406c7d0 2023-11-25T05:26:20.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.213+0000 7f6b0259c640 1 --2- 172.21.15.7:0/2816409301 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6ae406c7d0 0x7f6ae406ec90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6af4009ac0 tx=0x7f6af4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:20.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.214+0000 7f6afaffd640 1 -- 172.21.15.7:0/2816409301 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 1..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f6afc059b10 con 0x7f6b04143580 2023-11-25T05:26:20.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.214+0000 7f6afaffd640 10 client.21232.objecter ms_dispatch 0x7f6b04000b90 osd_map(918..918 src has 1..918) v4 2023-11-25T05:26:20.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.214+0000 7f6afaffd640 3 client.21232.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:20.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.214+0000 7f6afaffd640 3 client.21232.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:20.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afaffd640 10 client.21232.objecter checking command 1 2023-11-25T05:26:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afaffd640 1 --2- 172.21.15.7:0/2816409301 >> [v2:172.21.15.7:6824/2271997805,v1:172.21.15.7:6825/2271997805] conn(0x7f6ae4074050 0x7f6ae40764f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afaffd640 20 client.21232.objecter _get_session s=0x7f6ae4073980 osd=0 3 2023-11-25T05:26:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afaffd640 20 client.21232.objecter put_session s=0x7f6ae4073980 osd=0 3 2023-11-25T05:26:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afaffd640 15 client.21232.objecter _session_command_op_remove -1 1 2023-11-25T05:26:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afaffd640 20 client.21232.objecter _get_session s=0x7f6ae4073980 osd=0 3 2023-11-25T05:26:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afaffd640 20 client.21232.objecter get_session s=0x7f6ae4073980 osd=0 3 2023-11-25T05:26:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afaffd640 15 client.21232.objecter _session_command_op_assign 0 1 2023-11-25T05:26:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afaffd640 20 client.21232.objecter put_session s=0x7f6ae4073980 osd=0 4 2023-11-25T05:26:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afaffd640 10 client.21232.objecter _send_command 1 2023-11-25T05:26:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afaffd640 1 -- 172.21.15.7:0/2816409301 --> [v2:172.21.15.7:6824/2271997805,v1:172.21.15.7:6825/2271997805] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ae4076bc0 con 0x7f6ae4074050 2023-11-25T05:26:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afaffd640 20 client.21232.objecter dump_active .. 0 homeless 2023-11-25T05:26:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afbfff640 1 --2- 172.21.15.7:0/2816409301 >> [v2:172.21.15.7:6824/2271997805,v1:172.21.15.7:6825/2271997805] conn(0x7f6ae4074050 0x7f6ae40764f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:20.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afbfff640 10 monclient: get_auth_request con 0x7f6ae4074050 auth_method 0 2023-11-25T05:26:20.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afaffd640 1 -- 172.21.15.7:0/2816409301 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=918) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6afc055e50 con 0x7f6b04143580 2023-11-25T05:26:20.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.215+0000 7f6afaffd640 10 monclient: handle_get_version_reply finishing 1 version 918 2023-11-25T05:26:20.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.218+0000 7f6afbfff640 1 --2- 172.21.15.7:0/2816409301 >> [v2:172.21.15.7:6824/2271997805,v1:172.21.15.7:6825/2271997805] conn(0x7f6ae4074050 0x7f6ae40764f0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:20.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.218+0000 7f6afaffd640 10 client.21232.objecter ms_handle_connect 0x7f6ae4074050 2023-11-25T05:26:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.245+0000 7f6afaffd640 1 -- 172.21.15.7:0/2816409301 <== osd.0 v2:172.21.15.7:6824/2271997805 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f6ae4076bc0 con 0x7f6ae4074050 2023-11-25T05:26:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.245+0000 7f6afaffd640 10 client.21232.objecter ms_dispatch 0x7f6b04000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:26:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.245+0000 7f6afaffd640 10 client.21232.objecter _finish_command 1 = system:0 2023-11-25T05:26:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.245+0000 7f6afaffd640 20 client.21232.objecter put_session s=0x7f6ae4073980 osd=0 4 2023-11-25T05:26:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.245+0000 7f6afaffd640 15 client.21232.objecter _session_command_op_remove 0 1 2023-11-25T05:26:20.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.258+0000 7f6b08d5a640 10 client.21232.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-11-25T05:26:20.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.258+0000 7f6b08d5a640 15 client.21232.objecter _session_command_op_assign -1 2 2023-11-25T05:26:20.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.258+0000 7f6b08d5a640 20 client.21232.objecter _get_session s=0x7f6ae4073980 osd=0 3 2023-11-25T05:26:20.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.258+0000 7f6b08d5a640 20 client.21232.objecter put_session s=0x7f6ae4073980 osd=0 3 2023-11-25T05:26:20.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.258+0000 7f6b08d5a640 20 client.21232.objecter _get_session s=0x7f6ae4073980 osd=0 3 2023-11-25T05:26:20.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.258+0000 7f6b08d5a640 15 client.21232.objecter _session_command_op_remove -1 2 2023-11-25T05:26:20.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.258+0000 7f6b08d5a640 20 client.21232.objecter get_session s=0x7f6ae4073980 osd=0 3 2023-11-25T05:26:20.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.258+0000 7f6b08d5a640 15 client.21232.objecter _session_command_op_assign 0 2 2023-11-25T05:26:20.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.258+0000 7f6b08d5a640 20 client.21232.objecter put_session s=0x7f6ae4073980 osd=0 4 2023-11-25T05:26:20.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.258+0000 7f6b08d5a640 10 client.21232.objecter _send_command 2 2023-11-25T05:26:20.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.258+0000 7f6b08d5a640 1 -- 172.21.15.7:0/2816409301 --> [v2:172.21.15.7:6824/2271997805,v1:172.21.15.7:6825/2271997805] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6b040adc70 con 0x7f6ae4074050 2023-11-25T05:26:20.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6afaffd640 1 -- 172.21.15.7:0/2816409301 <== osd.0 v2:172.21.15.7:6824/2271997805 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6b040adc70 con 0x7f6ae4074050 2023-11-25T05:26:20.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6afaffd640 10 client.21232.objecter ms_dispatch 0x7f6b04000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:26:20.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6afaffd640 10 client.21232.objecter _finish_command 2 = system:0 2023-11-25T05:26:20.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6afaffd640 20 client.21232.objecter put_session s=0x7f6ae4073980 osd=0 4 2023-11-25T05:26:20.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6afaffd640 15 client.21232.objecter _session_command_op_remove 0 2 2023-11-25T05:26:20.275 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-11-25T05:26:20.275 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-25T05:26:20.275 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-11-25T05:26:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 10 librados: watch_flush enter 2023-11-25T05:26:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 10 librados: watch_flush exit 2023-11-25T05:26:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 10 client.21232.objecter close_session for osd.0 2023-11-25T05:26:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 1 -- 172.21.15.7:0/2816409301 >> [v2:172.21.15.7:6824/2271997805,v1:172.21.15.7:6825/2271997805] conn(0x7f6ae4074050 msgr2=0x7f6ae40764f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 1 --2- 172.21.15.7:0/2816409301 >> [v2:172.21.15.7:6824/2271997805,v1:172.21.15.7:6825/2271997805] conn(0x7f6ae4074050 0x7f6ae40764f0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 20 client.21232.objecter put_session s=0x7f6ae4073980 osd=0 1 2023-11-25T05:26:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 20 client.21232.objecter shutdown clearing up homeless session... 2023-11-25T05:26:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 10 client.21232.objecter successfully canceled tick 2023-11-25T05:26:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 1 -- 172.21.15.7:0/2816409301 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6ae406c7d0 msgr2=0x7f6ae406ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 1 --2- 172.21.15.7:0/2816409301 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6ae406c7d0 0x7f6ae406ec90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6af4009ac0 tx=0x7f6af4009190 comp rx=0 tx=0).stop 2023-11-25T05:26:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 10 monclient: shutdown 2023-11-25T05:26:20.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:20.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 1 -- 172.21.15.7:0/2816409301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b04143580 msgr2=0x7f6b040aa1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:20.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 1 --2- 172.21.15.7:0/2816409301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b04143580 0x7f6b040aa1e0 secure :-1 s=READY pgs=5149 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00efc0 tx=0x7f6afc00c490 comp rx=0 tx=0).stop 2023-11-25T05:26:20.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b02d9d640 1 -- 172.21.15.7:0/2816409301 reap_dead start 2023-11-25T05:26:20.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 1 -- 172.21.15.7:0/2816409301 shutdown_connections 2023-11-25T05:26:20.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.273+0000 7f6b08d5a640 1 -- 172.21.15.7:0/2816409301 >> 172.21.15.7:0/2816409301 conn(0x7f6b040b01e0 msgr2=0x7f6b0413c160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:20.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.274+0000 7f6b08d5a640 1 -- 172.21.15.7:0/2816409301 shutdown_connections 2023-11-25T05:26:20.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.274+0000 7f6b08d5a640 1 -- 172.21.15.7:0/2816409301 wait complete. 2023-11-25T05:26:20.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.274+0000 7f6b08d5a640 1 librados: shutdown 2023-11-25T05:26:20.289 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-11-25T05:26:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.411+0000 7f38d8eb4640 10 monclient: get_monmap_and_config 2023-11-25T05:26:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.411+0000 7f38d8eb4640 10 monclient: build_initial_monmap 2023-11-25T05:26:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.411+0000 7f38d8eb4640 10 monclient: monmap: 2023-11-25T05:26:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:20.412663+0000 2023-11-25T05:26:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:20.412663+0000 2023-11-25T05:26:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:20.414 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:20.414 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:20.414 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:20.414 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:20.414 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:20.414 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.412+0000 7f38d8eb4640 1 Processor -- start 2023-11-25T05:26:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.412+0000 7f38d8eb4640 1 -- start start 2023-11-25T05:26:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.412+0000 7f38d8eb4640 10 monclient: init 2023-11-25T05:26:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d8eb4640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d8eb4640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-11-25T05:26:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d8eb4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38d40afbf0 addr v1:172.21.15.90:6789/0 2023-11-25T05:26:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d8eb4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38d413bbb0 0x7f38d4070610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d8eb4640 10 monclient(hunting): picked mon.noname-b con 0x7f38d413bbb0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d8eb4640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f38d4071f10 0x7f38d40723a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d8eb4640 10 monclient(hunting): picked mon.noname-c con 0x7f38d4071f10 addr v2:172.21.15.90:3300/0 2023-11-25T05:26:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d8eb4640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d406f2e0 con 0x7f38d40afbf0 2023-11-25T05:26:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d8eb4640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d8eb4640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d40affd0 con 0x7f38d4071f10 2023-11-25T05:26:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d8eb4640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d8eb4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d406f5c0 con 0x7f38d413bbb0 2023-11-25T05:26:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d8eb4640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d8eb4640 10 monclient(hunting): authenticate will time out at 2706.656001s 2023-11-25T05:26:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d1d9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38d413bbb0 0x7f38d4070610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.413+0000 7f38d2d9d640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f38d4071f10 0x7f38d40723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d1d9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38d413bbb0 0x7f38d4070610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38232/0 (socket says 172.21.15.7:38232) 2023-11-25T05:26:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d1d9b640 1 -- 172.21.15.7:0/562146746 learned_addr learned my addr 172.21.15.7:0/562146746 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d1d9b640 10 monclient(hunting): get_auth_request con 0x7f38d413bbb0 auth_method 0 2023-11-25T05:26:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d1d9b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d1d9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d2d9d640 10 monclient(hunting): get_auth_request con 0x7f38d4071f10 auth_method 0 2023-11-25T05:26:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d2d9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.414+0000 7f38d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:20.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.416+0000 7f38d1d9b640 10 monclient(hunting): handle_auth_done global_id 17384 payload 274 2023-11-25T05:26:20.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.416+0000 7f38d1d9b640 1 -- 172.21.15.7:0/562146746 >> v2:172.21.15.90:3300/0 conn(0x7f38d4071f10 msgr2=0x7f38d40723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:20.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.416+0000 7f38d1d9b640 1 --2- 172.21.15.7:0/562146746 >> v2:172.21.15.90:3300/0 conn(0x7f38d4071f10 0x7f38d40723a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:20.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.416+0000 7f38d1d9b640 1 -- 172.21.15.7:0/562146746 >> v1:172.21.15.90:6789/0 conn(0x7f38d40afbf0 legacy=0x7f38d413b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:20.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.416+0000 7f38d1d9b640 10 monclient: _finish_hunting 0 2023-11-25T05:26:20.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.416+0000 7f38d1d9b640 1 monclient: found mon.noname-b 2023-11-25T05:26:20.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.416+0000 7f38d1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:20.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.416+0000 7f38d1d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:26:20.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.416+0000 7f38d1d9b640 1 -- 172.21.15.7:0/562146746 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d40728e0 con 0x7f38d413bbb0 2023-11-25T05:26:20.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.419+0000 7f38d1d9b640 1 --2- 172.21.15.7:0/562146746 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38d413bbb0 0x7f38d4070610 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009870 tx=0x7f38c802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19f1ea7d4aec5d0b server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:20.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d159a640 1 -- 172.21.15.7:0/562146746 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c8030840 con 0x7f38d413bbb0 2023-11-25T05:26:20.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:20.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:20.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d159a640 10 monclient: dump: 2023-11-25T05:26:20.426 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:20.426 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:20.426 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.426 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.426 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:20.426 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:20.427 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:20.427 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:20.427 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:20.427 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:20.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d159a640 10 monclient: _finish_auth 0 2023-11-25T05:26:20.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d159a640 10 monclient: _check_auth_tickets 2023-11-25T05:26:20.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:20.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d159a640 1 -- 172.21.15.7:0/562146746 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c80309a0 con 0x7f38d413bbb0 2023-11-25T05:26:20.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d159a640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:20.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d159a640 1 -- 172.21.15.7:0/562146746 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c8030d30 con 0x7f38d413bbb0 2023-11-25T05:26:20.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:20.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:26:20.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d159a640 10 monclient: dump: 2023-11-25T05:26:20.428 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:20.428 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:20.428 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.428 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.428 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:20.428 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:20.429 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:20.429 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:20.429 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:20.429 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:20.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d8eb4640 5 monclient: authenticate success, global_id 17384 2023-11-25T05:26:20.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d8eb4640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:20.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.424+0000 7f38d8eb4640 4 set_mon_vals no callback set 2023-11-25T05:26:20.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.425+0000 7f38d8eb4640 10 monclient: shutdown 2023-11-25T05:26:20.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.425+0000 7f38d8eb4640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:20.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.425+0000 7f38d8eb4640 1 -- 172.21.15.7:0/562146746 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38d413bbb0 msgr2=0x7f38d4070610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:20.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.425+0000 7f38d8eb4640 1 --2- 172.21.15.7:0/562146746 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38d413bbb0 0x7f38d4070610 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009870 tx=0x7f38c802b980 comp rx=0 tx=0).stop 2023-11-25T05:26:20.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.425+0000 7f38d8eb4640 1 -- 172.21.15.7:0/562146746 shutdown_connections 2023-11-25T05:26:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.425+0000 7f38d8eb4640 1 --2- 172.21.15.7:0/562146746 >> v2:172.21.15.90:3300/0 conn(0x7f38d4071f10 0x7f38d40723a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.425+0000 7f38d8eb4640 1 --2- 172.21.15.7:0/562146746 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38d413bbb0 0x7f38d4070610 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.425+0000 7f38d8eb4640 1 -- 172.21.15.7:0/562146746 >> 172.21.15.7:0/562146746 conn(0x7f38d40b01e0 msgr2=0x7f38d40b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.425+0000 7f38d8eb4640 1 -- 172.21.15.7:0/562146746 shutdown_connections 2023-11-25T05:26:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.425+0000 7f38d8eb4640 1 -- 172.21.15.7:0/562146746 wait complete. 2023-11-25T05:26:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.426+0000 7f38d8eb4640 10 monclient: build_initial_monmap 2023-11-25T05:26:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.426+0000 7f38d8eb4640 10 monclient: monmap: 2023-11-25T05:26:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:20.438 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:20.438 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:20.438 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:20.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.426+0000 7f38d8eb4640 1 librados: starting msgr at 2023-11-25T05:26:20.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.426+0000 7f38d8eb4640 1 librados: starting objecter 2023-11-25T05:26:20.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 1 Processor -- start 2023-11-25T05:26:20.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 1 -- start start 2023-11-25T05:26:20.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 1 librados: setting wanted keys 2023-11-25T05:26:20.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 1 librados: calling monclient init 2023-11-25T05:26:20.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 10 monclient: init 2023-11-25T05:26:20.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:20.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-25T05:26:20.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38d4071f10 0x7f38d40acb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:20.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 10 monclient(hunting): picked mon.noname-b con 0x7f38d4071f10 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:20.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38d40afbf0 0x7f38d413ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:20.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 10 monclient(hunting): picked mon.noname-c con 0x7f38d40afbf0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:20.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38d413bbb0 0x7f38d41da6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:20.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 10 monclient(hunting): picked mon.noname-a con 0x7f38d413bbb0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:20.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:20.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d413d720 con 0x7f38d413bbb0 2023-11-25T05:26:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d413c000 con 0x7f38d4071f10 2023-11-25T05:26:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d413dbb0 con 0x7f38d40afbf0 2023-11-25T05:26:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.427+0000 7f38d8eb4640 10 monclient(hunting): authenticate will time out at 2706.670479s 2023-11-25T05:26:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d1d9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38d40afbf0 0x7f38d413ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d2d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38d413bbb0 0x7f38d41da6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38d4071f10 0x7f38d40acb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d1d9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38d40afbf0 0x7f38d413ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38234/0 (socket says 172.21.15.7:38234) 2023-11-25T05:26:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d1d9b640 1 -- 172.21.15.7:0/3066432854 learned_addr learned my addr 172.21.15.7:0/3066432854 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d1d9b640 10 monclient(hunting): get_auth_request con 0x7f38d40afbf0 auth_method 0 2023-11-25T05:26:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d1d9b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d1d9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d259c640 10 monclient(hunting): get_auth_request con 0x7f38d4071f10 auth_method 0 2023-11-25T05:26:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d259c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d259c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.428+0000 7f38d1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.429+0000 7f38d1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.429+0000 7f38d2d9d640 10 monclient(hunting): get_auth_request con 0x7f38d413bbb0 auth_method 0 2023-11-25T05:26:20.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.429+0000 7f38d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:20.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.429+0000 7f38d2d9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:20.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.429+0000 7f38d2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:20.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.430+0000 7f38d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:20.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.430+0000 7f38d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:20.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.430+0000 7f38d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:20.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.431+0000 7f38d259c640 10 monclient(hunting): handle_auth_done global_id 21238 payload 824 2023-11-25T05:26:20.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.431+0000 7f38d259c640 1 -- 172.21.15.7:0/3066432854 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38d40afbf0 msgr2=0x7f38d413ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:20.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.431+0000 7f38d259c640 1 --2- 172.21.15.7:0/3066432854 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38d40afbf0 0x7f38d413ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:20.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.431+0000 7f38d259c640 1 -- 172.21.15.7:0/3066432854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38d413bbb0 msgr2=0x7f38d41da6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:20.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.431+0000 7f38d259c640 1 --2- 172.21.15.7:0/3066432854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38d413bbb0 0x7f38d41da6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:20.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.431+0000 7f38d259c640 10 monclient: _finish_hunting 0 2023-11-25T05:26:20.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.431+0000 7f38d259c640 1 monclient: found mon.noname-b 2023-11-25T05:26:20.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.431+0000 7f38d259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:20.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.431+0000 7f38d259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:26:20.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.431+0000 7f38d259c640 1 -- 172.21.15.7:0/3066432854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c80094d0 con 0x7f38d4071f10 2023-11-25T05:26:20.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.431+0000 7f38d1d9b640 1 --2- 172.21.15.7:0/3066432854 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38d40afbf0 0x7f38d413ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-25T05:26:20.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.433+0000 7f38d259c640 1 --2- 172.21.15.7:0/3066432854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38d4071f10 0x7f38d40acb50 secure :-1 s=READY pgs=5151 cs=0 l=1 rev1=1 crypto rx=0x7f38c400c370 tx=0x7f38c400c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:20.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.433+0000 7f38c2ffd640 10 client.?.objecter ms_handle_connect 0x7f38d4071f10 2023-11-25T05:26:20.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.433+0000 7f38c2ffd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38c2ffd640 1 -- 172.21.15.7:0/3066432854 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c40052d0 con 0x7f38d4071f10 2023-11-25T05:26:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38c2ffd640 10 monclient: dump: 2023-11-25T05:26:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:20.445 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38c2ffd640 10 monclient: _finish_auth 0 2023-11-25T05:26:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38c2ffd640 10 monclient: _check_auth_tickets 2023-11-25T05:26:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38c2ffd640 1 -- 172.21.15.7:0/3066432854 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c4005430 con 0x7f38d4071f10 2023-11-25T05:26:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38c2ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38c2ffd640 1 -- 172.21.15.7:0/3066432854 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c4003850 con 0x7f38d4071f10 2023-11-25T05:26:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38c2ffd640 10 monclient: dump: 2023-11-25T05:26:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:20.446 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.447 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:20.447 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:20.447 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:20.447 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:20.447 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:20.447 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:20.447 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:20.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38d8eb4640 5 monclient: authenticate success, global_id 21238 2023-11-25T05:26:20.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38d8eb4640 10 monclient: _renew_subs 2023-11-25T05:26:20.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38d8eb4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:20.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38d0d99640 4 set_mon_vals no callback set 2023-11-25T05:26:20.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38d8eb4640 1 -- 172.21.15.7:0/3066432854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d413d210 con 0x7f38d4071f10 2023-11-25T05:26:20.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38d8eb4640 10 client.21238.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:20.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38d8eb4640 10 monclient: _renew_subs 2023-11-25T05:26:20.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38d8eb4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:20.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38d8eb4640 1 -- 172.21.15.7:0/3066432854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d41dadc0 con 0x7f38d4071f10 2023-11-25T05:26:20.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.440+0000 7f38d8eb4640 1 librados: init done 2023-11-25T05:26:20.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.441+0000 7f38d8eb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:20.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.441+0000 7f38d8eb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:20.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.441+0000 7f38d8eb4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:20.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.441+0000 7f38d8eb4640 1 -- 172.21.15.7:0/3066432854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d4075b60 con 0x7f38d4071f10 2023-11-25T05:26:20.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.460+0000 7f38c2ffd640 1 -- 172.21.15.7:0/3066432854 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f38c40039b0 con 0x7f38d4071f10 2023-11-25T05:26:20.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.460+0000 7f38c2ffd640 1 --2- 172.21.15.7:0/3066432854 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f38b406c820 0x7f38b406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:20.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.461+0000 7f38d1d9b640 1 --2- 172.21.15.7:0/3066432854 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f38b406c820 0x7f38b406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:20.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.461+0000 7f38d1d9b640 10 monclient: get_auth_request con 0x7f38b406c820 auth_method 0 2023-11-25T05:26:20.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.461+0000 7f38c2ffd640 10 client.21238.objecter ms_handle_connect 0x7f38b406c820 2023-11-25T05:26:20.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.461+0000 7f38d1d9b640 1 --2- 172.21.15.7:0/3066432854 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f38b406c820 0x7f38b406ece0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009870 tx=0x7f38c8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:20.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.463+0000 7f38c2ffd640 1 -- 172.21.15.7:0/3066432854 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 1..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f38c4057e50 con 0x7f38d4071f10 2023-11-25T05:26:20.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.463+0000 7f38c2ffd640 10 client.21238.objecter ms_dispatch 0x7f38d4000b90 osd_map(918..918 src has 1..918) v4 2023-11-25T05:26:20.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.463+0000 7f38c2ffd640 3 client.21238.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:20.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.463+0000 7f38c2ffd640 3 client.21238.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:20.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.463+0000 7f38c2ffd640 20 client.21238.objecter dump_active .. 0 homeless 2023-11-25T05:26:20.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.698+0000 7f38c2ffd640 1 -- 172.21.15.7:0/3066432854 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38c4054190 con 0x7f38d4071f10 2023-11-25T05:26:20.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.698+0000 7f38c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:20.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.698+0000 7f38c2ffd640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:26:20.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.878+0000 7f38d8eb4640 1 -- 172.21.15.7:0/3066432854 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f38d40af0f0 con 0x7f38b406c820 2023-11-25T05:26:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.878+0000 7f38c2ffd640 1 -- 172.21.15.7:0/3066432854 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f38c8039550 con 0x7f38b406c820 2023-11-25T05:26:21.035 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 0 to deep-scrub 2023-11-25T05:26:21.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.880+0000 7f38d8eb4640 10 librados: watch_flush enter 2023-11-25T05:26:21.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.880+0000 7f38d8eb4640 10 librados: watch_flush exit 2023-11-25T05:26:21.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.880+0000 7f38d8eb4640 20 client.21238.objecter shutdown clearing up homeless session... 2023-11-25T05:26:21.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.880+0000 7f38d8eb4640 10 client.21238.objecter successfully canceled tick 2023-11-25T05:26:21.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.881+0000 7f38d8eb4640 1 -- 172.21.15.7:0/3066432854 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f38b406c820 msgr2=0x7f38b406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:21.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.881+0000 7f38d8eb4640 1 --2- 172.21.15.7:0/3066432854 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f38b406c820 0x7f38b406ece0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009870 tx=0x7f38c8008030 comp rx=0 tx=0).stop 2023-11-25T05:26:21.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.881+0000 7f38d8eb4640 10 monclient: shutdown 2023-11-25T05:26:21.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.881+0000 7f38d8eb4640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:21.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.881+0000 7f38d8eb4640 1 -- 172.21.15.7:0/3066432854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38d4071f10 msgr2=0x7f38d40acb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:21.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.881+0000 7f38d8eb4640 1 --2- 172.21.15.7:0/3066432854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38d4071f10 0x7f38d40acb50 secure :-1 s=READY pgs=5151 cs=0 l=1 rev1=1 crypto rx=0x7f38c400c370 tx=0x7f38c400c830 comp rx=0 tx=0).stop 2023-11-25T05:26:21.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.881+0000 7f38d8eb4640 1 -- 172.21.15.7:0/3066432854 shutdown_connections 2023-11-25T05:26:21.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.881+0000 7f38d8eb4640 1 --2- 172.21.15.7:0/3066432854 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f38b406c820 0x7f38b406ece0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.881+0000 7f38d8eb4640 1 --2- 172.21.15.7:0/3066432854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38d413bbb0 0x7f38d41da6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.881+0000 7f38d8eb4640 1 --2- 172.21.15.7:0/3066432854 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f38d40afbf0 0x7f38d413ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.881+0000 7f38d8eb4640 1 --2- 172.21.15.7:0/3066432854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38d4071f10 0x7f38d40acb50 unknown :-1 s=CLOSED pgs=5151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.881+0000 7f38d8eb4640 1 -- 172.21.15.7:0/3066432854 >> 172.21.15.7:0/3066432854 conn(0x7f38d40b01e0 msgr2=0x7f38d406eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.881+0000 7f38d8eb4640 1 -- 172.21.15.7:0/3066432854 shutdown_connections 2023-11-25T05:26:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.881+0000 7f38d8eb4640 1 -- 172.21.15.7:0/3066432854 wait complete. 2023-11-25T05:26:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:20.881+0000 7f38d8eb4640 1 librados: shutdown 2023-11-25T05:26:21.038 INFO:tasks.ceph:Scrubbing osd.1 2023-11-25T05:26:21.039 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-11-25T05:26:21.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.157+0000 7f950686f640 10 monclient: get_monmap_and_config 2023-11-25T05:26:21.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.157+0000 7f950686f640 10 monclient: build_initial_monmap 2023-11-25T05:26:21.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.157+0000 7f950686f640 10 monclient: monmap: 2023-11-25T05:26:21.158 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:21.159 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:21.159 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:21.158105+0000 2023-11-25T05:26:21.159 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:21.158105+0000 2023-11-25T05:26:21.159 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:21.159 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:21.159 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:21.159 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:21.159 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:21.159 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:21.159 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:21.159 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:21.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.157+0000 7f950686f640 1 Processor -- start 2023-11-25T05:26:21.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 1 -- start start 2023-11-25T05:26:21.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 10 monclient: init 2023-11-25T05:26:21.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:21.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-11-25T05:26:21.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95000711b0 0x7f9500071590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:21.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 10 monclient(hunting): picked mon.noname-b con 0x7f95000711b0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:21.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9500070400 addr v1:172.21.15.90:6789/0 2023-11-25T05:26:21.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f950006c240 0x7f950006c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:21.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 10 monclient(hunting): picked mon.noname-a con 0x7f950006c240 addr v2:172.21.15.7:3300/0 2023-11-25T05:26:21.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95000afca0 con 0x7f9500070400 2023-11-25T05:26:21.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:21.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f950006be20 con 0x7f950006c240 2023-11-25T05:26:21.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:21.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950013f9c0 con 0x7f95000711b0 2023-11-25T05:26:21.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:21.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.158+0000 7f950686f640 10 monclient(hunting): authenticate will time out at 2707.401563s 2023-11-25T05:26:21.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.159+0000 7f94fffff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95000711b0 0x7f9500071590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:21.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.159+0000 7f94fffff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95000711b0 0x7f9500071590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38248/0 (socket says 172.21.15.7:38248) 2023-11-25T05:26:21.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.159+0000 7f94fffff640 1 -- 172.21.15.7:0/3989744726 learned_addr learned my addr 172.21.15.7:0/3989744726 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:21.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.159+0000 7f94fffff640 10 monclient(hunting): get_auth_request con 0x7f95000711b0 auth_method 0 2023-11-25T05:26:21.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.159+0000 7f94fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:21.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.159+0000 7f94ff7fe640 1 --1- 172.21.15.7:0/3989744726 >> v1:172.21.15.90:6789/0 conn(0x7f9500070400 0x7f950013f2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:40516/0 (socket says 172.21.15.7:40516) 2023-11-25T05:26:21.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.159+0000 7f94fffff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:21.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.159+0000 7f94fffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:21.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.160+0000 7f94fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:21.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.160+0000 7f94fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:21.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.160+0000 7f94fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:21.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.161+0000 7f94fffff640 10 monclient(hunting): handle_auth_done global_id 17390 payload 274 2023-11-25T05:26:21.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.161+0000 7f94fffff640 1 -- 172.21.15.7:0/3989744726 >> v2:172.21.15.7:3300/0 conn(0x7f950006c240 msgr2=0x7f950006c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:21.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.161+0000 7f94fffff640 1 --2- 172.21.15.7:0/3989744726 >> v2:172.21.15.7:3300/0 conn(0x7f950006c240 0x7f950006c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.161+0000 7f94fffff640 1 -- 172.21.15.7:0/3989744726 >> v1:172.21.15.90:6789/0 conn(0x7f9500070400 legacy=0x7f950013f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:21.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.161+0000 7f94fffff640 10 monclient: _finish_hunting 0 2023-11-25T05:26:21.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.161+0000 7f94fffff640 1 monclient: found mon.noname-b 2023-11-25T05:26:21.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.161+0000 7f94fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:21.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.161+0000 7f94fffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:26:21.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.161+0000 7f94fffff640 1 -- 172.21.15.7:0/3989744726 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950006cbf0 con 0x7f95000711b0 2023-11-25T05:26:21.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.163+0000 7f94fffff640 1 --2- 172.21.15.7:0/3989744726 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95000711b0 0x7f9500071590 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7f94f0009870 tx=0x7f94f002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e7a949b497c516e server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:21.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f94feffd640 1 -- 172.21.15.7:0/3989744726 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f0030840 con 0x7f95000711b0 2023-11-25T05:26:21.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f94feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:21.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f94feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:21.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f94feffd640 10 monclient: dump: 2023-11-25T05:26:21.169 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:21.169 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:21.170 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.170 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.170 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:21.170 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:21.170 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:21.170 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:21.170 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:21.170 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:21.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f94feffd640 10 monclient: _finish_auth 0 2023-11-25T05:26:21.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f94feffd640 10 monclient: _check_auth_tickets 2023-11-25T05:26:21.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f94feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f94feffd640 1 -- 172.21.15.7:0/3989744726 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f00309a0 con 0x7f95000711b0 2023-11-25T05:26:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f950686f640 5 monclient: authenticate success, global_id 17390 2023-11-25T05:26:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f950686f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:26:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f94feffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f94feffd640 1 -- 172.21.15.7:0/3989744726 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f0030d30 con 0x7f95000711b0 2023-11-25T05:26:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f950686f640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f950686f640 4 set_mon_vals no callback set 2023-11-25T05:26:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f94feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f94feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:26:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f94feffd640 10 monclient: dump: 2023-11-25T05:26:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:21.172 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:21.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f950686f640 10 monclient: shutdown 2023-11-25T05:26:21.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f950686f640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:21.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f950686f640 1 -- 172.21.15.7:0/3989744726 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95000711b0 msgr2=0x7f9500071590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:21.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.168+0000 7f950686f640 1 --2- 172.21.15.7:0/3989744726 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95000711b0 0x7f9500071590 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7f94f0009870 tx=0x7f94f002b980 comp rx=0 tx=0).stop 2023-11-25T05:26:21.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.169+0000 7f950686f640 1 -- 172.21.15.7:0/3989744726 shutdown_connections 2023-11-25T05:26:21.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.169+0000 7f950686f640 1 --2- 172.21.15.7:0/3989744726 >> v2:172.21.15.7:3300/0 conn(0x7f950006c240 0x7f950006c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.169+0000 7f950686f640 1 --2- 172.21.15.7:0/3989744726 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95000711b0 0x7f9500071590 unknown :-1 s=CLOSED pgs=3065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.169+0000 7f950686f640 1 -- 172.21.15.7:0/3989744726 >> 172.21.15.7:0/3989744726 conn(0x7f95000b5880 msgr2=0x7f95000b5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.169+0000 7f950686f640 1 -- 172.21.15.7:0/3989744726 shutdown_connections 2023-11-25T05:26:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.169+0000 7f950686f640 1 -- 172.21.15.7:0/3989744726 wait complete. 2023-11-25T05:26:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.169+0000 7f950686f640 10 monclient: build_initial_monmap 2023-11-25T05:26:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.169+0000 7f950686f640 10 monclient: monmap: 2023-11-25T05:26:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:21.177 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.170+0000 7f950686f640 1 librados: starting msgr at 2023-11-25T05:26:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.170+0000 7f950686f640 1 librados: starting objecter 2023-11-25T05:26:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.170+0000 7f950686f640 1 Processor -- start 2023-11-25T05:26:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.170+0000 7f950686f640 1 -- start start 2023-11-25T05:26:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.170+0000 7f950686f640 1 librados: setting wanted keys 2023-11-25T05:26:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.170+0000 7f950686f640 1 librados: calling monclient init 2023-11-25T05:26:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.170+0000 7f950686f640 10 monclient: init 2023-11-25T05:26:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-25T05:26:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f950006c240 0x7f95000b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 10 monclient(hunting): picked mon.noname-b con 0x7f950006c240 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9500070400 0x7f95001378b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 10 monclient(hunting): picked mon.noname-a con 0x7f9500070400 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95000711b0 0x7f9500142200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 10 monclient(hunting): picked mon.noname-c con 0x7f95000711b0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950013fd10 con 0x7f9500070400 2023-11-25T05:26:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9500138310 con 0x7f950006c240 2023-11-25T05:26:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9500138a90 con 0x7f95000711b0 2023-11-25T05:26:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.171+0000 7f950686f640 10 monclient(hunting): authenticate will time out at 2707.414570s 2023-11-25T05:26:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.172+0000 7f9504e0c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95000711b0 0x7f9500142200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.172+0000 7f9504e0c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95000711b0 0x7f9500142200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38264/0 (socket says 172.21.15.7:38264) 2023-11-25T05:26:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.172+0000 7f9504e0c640 1 -- 172.21.15.7:0/1832890217 learned_addr learned my addr 172.21.15.7:0/1832890217 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.172+0000 7f9504e0c640 10 monclient(hunting): get_auth_request con 0x7f95000711b0 auth_method 0 2023-11-25T05:26:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.172+0000 7f9504e0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.172+0000 7f9504e0c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.172+0000 7f9504e0c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.172+0000 7f94fffff640 1 --2- 172.21.15.7:0/1832890217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f950006c240 0x7f95000b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.172+0000 7f94fffff640 10 monclient(hunting): get_auth_request con 0x7f950006c240 auth_method 0 2023-11-25T05:26:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.172+0000 7f94fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.172+0000 7f94fffff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.172+0000 7f94fffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.173+0000 7f9504e0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.173+0000 7f9504e0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.173+0000 7f9504e0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.173+0000 7f94fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.173+0000 7f94fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.173+0000 7f94fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:21.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.176+0000 7f9504e0c640 10 monclient(hunting): handle_auth_done global_id 17393 payload 824 2023-11-25T05:26:21.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.176+0000 7f9504e0c640 1 -- 172.21.15.7:0/1832890217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f950006c240 msgr2=0x7f95000b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:21.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.176+0000 7f9504e0c640 1 --2- 172.21.15.7:0/1832890217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f950006c240 0x7f95000b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.176+0000 7f9504e0c640 1 -- 172.21.15.7:0/1832890217 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9500070400 msgr2=0x7f95001378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:21.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.176+0000 7f9504e0c640 1 --2- 172.21.15.7:0/1832890217 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9500070400 0x7f95001378b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.176+0000 7f9504e0c640 10 monclient: _finish_hunting 0 2023-11-25T05:26:21.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.176+0000 7f9504e0c640 1 monclient: found mon.noname-c 2023-11-25T05:26:21.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.176+0000 7f9504e0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:21.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.176+0000 7f9504e0c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:26:21.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.176+0000 7f9504e0c640 1 -- 172.21.15.7:0/1832890217 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f00094d0 con 0x7f95000711b0 2023-11-25T05:26:21.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.178+0000 7f9504e0c640 1 --2- 172.21.15.7:0/1832890217 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95000711b0 0x7f9500142200 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f94f800b970 tx=0x7f94f800be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:21.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.178+0000 7f94fcff9640 10 client.?.objecter ms_handle_connect 0x7f95000711b0 2023-11-25T05:26:21.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.178+0000 7f94fcff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:21.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f94fcff9640 1 -- 172.21.15.7:0/1832890217 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f800cb40 con 0x7f95000711b0 2023-11-25T05:26:21.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f94fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:21.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f94fcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:26:21.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f94fcff9640 10 monclient: dump: 2023-11-25T05:26:21.184 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:21.184 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:21.184 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.184 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.184 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:21.184 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:21.184 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:21.184 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:21.184 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:21.184 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:21.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f94fcff9640 10 monclient: _finish_auth 0 2023-11-25T05:26:21.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f94fcff9640 10 monclient: _check_auth_tickets 2023-11-25T05:26:21.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f94fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:21.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f950686f640 5 monclient: authenticate success, global_id 17393 2023-11-25T05:26:21.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f94fcff9640 1 -- 172.21.15.7:0/1832890217 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f800ccc0 con 0x7f95000711b0 2023-11-25T05:26:21.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f94fcff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:21.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f94fcff9640 1 -- 172.21.15.7:0/1832890217 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f8014680 con 0x7f95000711b0 2023-11-25T05:26:21.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f94fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:21.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f94fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:26:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f94fcff9640 10 monclient: dump: 2023-11-25T05:26:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:21.186 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f94fe7fc640 4 set_mon_vals no callback set 2023-11-25T05:26:21.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f950686f640 10 monclient: _renew_subs 2023-11-25T05:26:21.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f950686f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:26:21.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f950686f640 1 -- 172.21.15.7:0/1832890217 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9500137df0 con 0x7f95000711b0 2023-11-25T05:26:21.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f950686f640 10 client.17393.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:21.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f950686f640 10 monclient: _renew_subs 2023-11-25T05:26:21.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f950686f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:26:21.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f950686f640 1 -- 172.21.15.7:0/1832890217 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9500142920 con 0x7f95000711b0 2023-11-25T05:26:21.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.182+0000 7f950686f640 1 librados: init done 2023-11-25T05:26:21.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.183+0000 7f950686f640 10 client.17393.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:21.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.183+0000 7f950686f640 15 client.17393.objecter _session_command_op_assign -1 1 2023-11-25T05:26:21.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.183+0000 7f950686f640 20 client.17393.objecter _get_session osd=-1 returning homeless 2023-11-25T05:26:21.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.183+0000 7f950686f640 10 client.17393.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:21.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.183+0000 7f950686f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:26:21.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.183+0000 7f950686f640 1 -- 172.21.15.7:0/1832890217 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9500070d90 con 0x7f95000711b0 2023-11-25T05:26:21.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.203+0000 7f94fcff9640 1 -- 172.21.15.7:0/1832890217 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f94f801d3b0 con 0x7f95000711b0 2023-11-25T05:26:21.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.204+0000 7f94fcff9640 1 --2- 172.21.15.7:0/1832890217 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f94ec06c7d0 0x7f94ec06ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:21.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.204+0000 7f94fffff640 1 --2- 172.21.15.7:0/1832890217 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f94ec06c7d0 0x7f94ec06ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:21.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.204+0000 7f94fffff640 10 monclient: get_auth_request con 0x7f94ec06c7d0 auth_method 0 2023-11-25T05:26:21.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.205+0000 7f94fcff9640 10 client.17393.objecter ms_handle_connect 0x7f94ec06c7d0 2023-11-25T05:26:21.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.205+0000 7f94fffff640 1 --2- 172.21.15.7:0/1832890217 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f94ec06c7d0 0x7f94ec06ec90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f94f0036000 tx=0x7f94f003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:21.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 1 -- 172.21.15.7:0/1832890217 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(918..918 src has 1..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f94f800ce30 con 0x7f95000711b0 2023-11-25T05:26:21.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 10 client.17393.objecter ms_dispatch 0x7f9500000b90 osd_map(918..918 src has 1..918) v4 2023-11-25T05:26:21.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 3 client.17393.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:21.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 3 client.17393.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:21.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 10 client.17393.objecter checking command 1 2023-11-25T05:26:21.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 1 --2- 172.21.15.7:0/1832890217 >> [v2:172.21.15.7:6800/3792760010,v1:172.21.15.7:6801/3792760010] conn(0x7f94ec074050 0x7f94ec0764f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:21.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 20 client.17393.objecter _get_session s=0x7f94ec073980 osd=1 3 2023-11-25T05:26:21.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 20 client.17393.objecter put_session s=0x7f94ec073980 osd=1 3 2023-11-25T05:26:21.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 15 client.17393.objecter _session_command_op_remove -1 1 2023-11-25T05:26:21.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 20 client.17393.objecter _get_session s=0x7f94ec073980 osd=1 3 2023-11-25T05:26:21.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 20 client.17393.objecter get_session s=0x7f94ec073980 osd=1 3 2023-11-25T05:26:21.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 15 client.17393.objecter _session_command_op_assign 1 1 2023-11-25T05:26:21.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 20 client.17393.objecter put_session s=0x7f94ec073980 osd=1 4 2023-11-25T05:26:21.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 10 client.17393.objecter _send_command 1 2023-11-25T05:26:21.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 1 -- 172.21.15.7:0/1832890217 --> [v2:172.21.15.7:6800/3792760010,v1:172.21.15.7:6801/3792760010] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f94ec076bc0 con 0x7f94ec074050 2023-11-25T05:26:21.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.207+0000 7f94fcff9640 20 client.17393.objecter dump_active .. 0 homeless 2023-11-25T05:26:21.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.208+0000 7f94ff7fe640 1 --2- 172.21.15.7:0/1832890217 >> [v2:172.21.15.7:6800/3792760010,v1:172.21.15.7:6801/3792760010] conn(0x7f94ec074050 0x7f94ec0764f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:21.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.208+0000 7f94ff7fe640 10 monclient: get_auth_request con 0x7f94ec074050 auth_method 0 2023-11-25T05:26:21.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.209+0000 7f94fcff9640 1 -- 172.21.15.7:0/1832890217 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_get_version_reply(handle=1 version=918) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94f8054d80 con 0x7f95000711b0 2023-11-25T05:26:21.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.209+0000 7f94fcff9640 10 monclient: handle_get_version_reply finishing 1 version 918 2023-11-25T05:26:21.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.211+0000 7f94fcff9640 10 client.17393.objecter ms_handle_connect 0x7f94ec074050 2023-11-25T05:26:21.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.211+0000 7f94ff7fe640 1 --2- 172.21.15.7:0/1832890217 >> [v2:172.21.15.7:6800/3792760010,v1:172.21.15.7:6801/3792760010] conn(0x7f94ec074050 0x7f94ec0764f0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:21.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.238+0000 7f94fcff9640 1 -- 172.21.15.7:0/1832890217 <== osd.1 v2:172.21.15.7:6800/3792760010 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f94ec076bc0 con 0x7f94ec074050 2023-11-25T05:26:21.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.238+0000 7f94fcff9640 10 client.17393.objecter ms_dispatch 0x7f9500000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:26:21.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.238+0000 7f94fcff9640 10 client.17393.objecter _finish_command 1 = system:0 2023-11-25T05:26:21.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.238+0000 7f94fcff9640 20 client.17393.objecter put_session s=0x7f94ec073980 osd=1 4 2023-11-25T05:26:21.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.238+0000 7f94fcff9640 15 client.17393.objecter _session_command_op_remove 1 1 2023-11-25T05:26:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.251+0000 7f950686f640 10 client.17393.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-11-25T05:26:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.251+0000 7f950686f640 15 client.17393.objecter _session_command_op_assign -1 2 2023-11-25T05:26:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.251+0000 7f950686f640 20 client.17393.objecter _get_session s=0x7f94ec073980 osd=1 3 2023-11-25T05:26:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.251+0000 7f950686f640 20 client.17393.objecter put_session s=0x7f94ec073980 osd=1 3 2023-11-25T05:26:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.251+0000 7f950686f640 20 client.17393.objecter _get_session s=0x7f94ec073980 osd=1 3 2023-11-25T05:26:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.251+0000 7f950686f640 15 client.17393.objecter _session_command_op_remove -1 2 2023-11-25T05:26:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.251+0000 7f950686f640 20 client.17393.objecter get_session s=0x7f94ec073980 osd=1 3 2023-11-25T05:26:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.251+0000 7f950686f640 15 client.17393.objecter _session_command_op_assign 1 2 2023-11-25T05:26:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.251+0000 7f950686f640 20 client.17393.objecter put_session s=0x7f94ec073980 osd=1 4 2023-11-25T05:26:21.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.251+0000 7f950686f640 10 client.17393.objecter _send_command 2 2023-11-25T05:26:21.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.251+0000 7f950686f640 1 -- 172.21.15.7:0/1832890217 --> [v2:172.21.15.7:6800/3792760010,v1:172.21.15.7:6801/3792760010] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9500075af0 con 0x7f94ec074050 2023-11-25T05:26:21.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f94fcff9640 1 -- 172.21.15.7:0/1832890217 <== osd.1 v2:172.21.15.7:6800/3792760010 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9500075af0 con 0x7f94ec074050 2023-11-25T05:26:21.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f94fcff9640 10 client.17393.objecter ms_dispatch 0x7f9500000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:26:21.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f94fcff9640 10 client.17393.objecter _finish_command 2 = system:0 2023-11-25T05:26:21.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f94fcff9640 20 client.17393.objecter put_session s=0x7f94ec073980 osd=1 4 2023-11-25T05:26:21.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f94fcff9640 15 client.17393.objecter _session_command_op_remove 1 2 2023-11-25T05:26:21.268 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-11-25T05:26:21.268 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-25T05:26:21.268 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-11-25T05:26:21.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f950686f640 10 librados: watch_flush enter 2023-11-25T05:26:21.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f950686f640 10 librados: watch_flush exit 2023-11-25T05:26:21.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f950686f640 10 client.17393.objecter close_session for osd.1 2023-11-25T05:26:21.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f950686f640 1 -- 172.21.15.7:0/1832890217 >> [v2:172.21.15.7:6800/3792760010,v1:172.21.15.7:6801/3792760010] conn(0x7f94ec074050 msgr2=0x7f94ec0764f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:21.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f950686f640 1 --2- 172.21.15.7:0/1832890217 >> [v2:172.21.15.7:6800/3792760010,v1:172.21.15.7:6801/3792760010] conn(0x7f94ec074050 0x7f94ec0764f0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f950686f640 20 client.17393.objecter put_session s=0x7f94ec073980 osd=1 1 2023-11-25T05:26:21.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f950686f640 20 client.17393.objecter shutdown clearing up homeless session... 2023-11-25T05:26:21.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f950686f640 10 client.17393.objecter successfully canceled tick 2023-11-25T05:26:21.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f950686f640 1 -- 172.21.15.7:0/1832890217 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f94ec06c7d0 msgr2=0x7f94ec06ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:21.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f950686f640 1 --2- 172.21.15.7:0/1832890217 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f94ec06c7d0 0x7f94ec06ec90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f94f0036000 tx=0x7f94f003b040 comp rx=0 tx=0).stop 2023-11-25T05:26:21.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f950686f640 10 monclient: shutdown 2023-11-25T05:26:21.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f950686f640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:21.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f950686f640 1 -- 172.21.15.7:0/1832890217 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95000711b0 msgr2=0x7f9500142200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:21.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.266+0000 7f950686f640 1 --2- 172.21.15.7:0/1832890217 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95000711b0 0x7f9500142200 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f94f800b970 tx=0x7f94f800be30 comp rx=0 tx=0).stop 2023-11-25T05:26:21.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.267+0000 7f9504e0c640 1 -- 172.21.15.7:0/1832890217 reap_dead start 2023-11-25T05:26:21.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.267+0000 7f950686f640 1 -- 172.21.15.7:0/1832890217 shutdown_connections 2023-11-25T05:26:21.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.267+0000 7f950686f640 1 -- 172.21.15.7:0/1832890217 >> 172.21.15.7:0/1832890217 conn(0x7f95000b5880 msgr2=0x7f9500135190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:21.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.267+0000 7f950686f640 1 -- 172.21.15.7:0/1832890217 shutdown_connections 2023-11-25T05:26:21.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.267+0000 7f950686f640 1 -- 172.21.15.7:0/1832890217 wait complete. 2023-11-25T05:26:21.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.267+0000 7f950686f640 1 librados: shutdown 2023-11-25T05:26:21.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-11-25T05:26:21.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.390+0000 7f33ce0e0640 10 monclient: get_monmap_and_config 2023-11-25T05:26:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.390+0000 7f33ce0e0640 10 monclient: build_initial_monmap 2023-11-25T05:26:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.390+0000 7f33ce0e0640 10 monclient: monmap: 2023-11-25T05:26:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:21.391311+0000 2023-11-25T05:26:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:21.391311+0000 2023-11-25T05:26:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:21.393 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:21.393 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:21.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 1 Processor -- start 2023-11-25T05:26:21.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 1 -- start start 2023-11-25T05:26:21.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 10 monclient: init 2023-11-25T05:26:21.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:21.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-11-25T05:26:21.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33c80a8910 0x7f33c813ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:21.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 10 monclient(hunting): picked mon.noname-b con 0x7f33c80a8910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:21.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f33c80a9b70 addr v1:172.21.15.90:6789/0 2023-11-25T05:26:21.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f33c8141f90 0x7f33c81423b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:21.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 10 monclient(hunting): picked mon.noname-c con 0x7f33c8141f90 addr v2:172.21.15.90:3300/0 2023-11-25T05:26:21.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c80a8cf0 con 0x7f33c80a9b70 2023-11-25T05:26:21.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:21.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c80aabd0 con 0x7f33c8141f90 2023-11-25T05:26:21.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:21.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c80aaeb0 con 0x7f33c80a8910 2023-11-25T05:26:21.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:21.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.391+0000 7f33ce0e0640 10 monclient(hunting): authenticate will time out at 2707.634740s 2023-11-25T05:26:21.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.392+0000 7f33c7fff640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f33c8141f90 0x7f33c81423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:21.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.392+0000 7f33c7fff640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f33c8141f90 0x7f33c81423b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:39302/0 (socket says 172.21.15.7:39302) 2023-11-25T05:26:21.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.392+0000 7f33c7fff640 1 -- 172.21.15.7:0/1801227655 learned_addr learned my addr 172.21.15.7:0/1801227655 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:21.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.392+0000 7f33c7fff640 10 monclient(hunting): get_auth_request con 0x7f33c8141f90 auth_method 0 2023-11-25T05:26:21.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.392+0000 7f33c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:21.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.392+0000 7f33c7fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:21.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.392+0000 7f33c7fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:21.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.393+0000 7f33c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:21.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.393+0000 7f33c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:21.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.393+0000 7f33c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:21.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.395+0000 7f33c7fff640 10 monclient(hunting): handle_auth_done global_id 21247 payload 274 2023-11-25T05:26:21.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.395+0000 7f33c7fff640 1 -- 172.21.15.7:0/1801227655 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33c80a8910 msgr2=0x7f33c813ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:21.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.395+0000 7f33c7fff640 1 --2- 172.21.15.7:0/1801227655 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33c80a8910 0x7f33c813ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.395+0000 7f33c7fff640 1 -- 172.21.15.7:0/1801227655 >> v1:172.21.15.90:6789/0 conn(0x7f33c80a9b70 legacy=0x7f33c813f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:21.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.395+0000 7f33c7fff640 10 monclient: _finish_hunting 0 2023-11-25T05:26:21.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.395+0000 7f33c7fff640 1 monclient: found mon.noname-c 2023-11-25T05:26:21.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.395+0000 7f33c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:21.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.395+0000 7f33c7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:26:21.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.395+0000 7f33c7fff640 1 -- 172.21.15.7:0/1801227655 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c80a9720 con 0x7f33c8141f90 2023-11-25T05:26:21.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.397+0000 7f33c7fff640 1 --2- 172.21.15.7:0/1801227655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33c8141f90 0x7f33c81423b0 secure :-1 s=READY pgs=5154 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009e50 tx=0x7f33c002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfb386c867e0f4ff server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:21.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33c67fc640 1 -- 172.21.15.7:0/1801227655 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c002ca70 con 0x7f33c8141f90 2023-11-25T05:26:21.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:21.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:26:21.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33c67fc640 10 monclient: dump: 2023-11-25T05:26:21.407 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:21.407 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:21.407 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.407 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.407 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:21.407 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:21.407 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:21.407 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:21.407 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:21.407 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:21.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33c67fc640 10 monclient: _finish_auth 0 2023-11-25T05:26:21.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33c67fc640 10 monclient: _check_auth_tickets 2023-11-25T05:26:21.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:21.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33c67fc640 1 -- 172.21.15.7:0/1801227655 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c002cbd0 con 0x7f33c8141f90 2023-11-25T05:26:21.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33c67fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:21.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33c67fc640 1 -- 172.21.15.7:0/1801227655 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c00343f0 con 0x7f33c8141f90 2023-11-25T05:26:21.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33ce0e0640 5 monclient: authenticate success, global_id 21247 2023-11-25T05:26:21.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33ce0e0640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:21.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33ce0e0640 4 set_mon_vals no callback set 2023-11-25T05:26:21.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:21.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:21.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33c67fc640 10 monclient: dump: 2023-11-25T05:26:21.408 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:21.408 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:21.408 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.409 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.409 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:21.409 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:21.409 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:21.409 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:21.409 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:21.409 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:21.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.405+0000 7f33ce0e0640 10 monclient: shutdown 2023-11-25T05:26:21.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.406+0000 7f33ce0e0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:21.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.406+0000 7f33ce0e0640 1 -- 172.21.15.7:0/1801227655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33c8141f90 msgr2=0x7f33c81423b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:21.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.406+0000 7f33ce0e0640 1 --2- 172.21.15.7:0/1801227655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33c8141f90 0x7f33c81423b0 secure :-1 s=READY pgs=5154 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009e50 tx=0x7f33c002bed0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.406+0000 7f33ce0e0640 1 -- 172.21.15.7:0/1801227655 shutdown_connections 2023-11-25T05:26:21.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.406+0000 7f33ce0e0640 1 --2- 172.21.15.7:0/1801227655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33c8141f90 0x7f33c81423b0 unknown :-1 s=CLOSED pgs=5154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.406+0000 7f33ce0e0640 1 --2- 172.21.15.7:0/1801227655 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33c80a8910 0x7f33c813ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.406+0000 7f33ce0e0640 1 -- 172.21.15.7:0/1801227655 >> 172.21.15.7:0/1801227655 conn(0x7f33c80b7780 msgr2=0x7f33c80b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:21.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.406+0000 7f33ce0e0640 1 -- 172.21.15.7:0/1801227655 shutdown_connections 2023-11-25T05:26:21.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.406+0000 7f33ce0e0640 1 -- 172.21.15.7:0/1801227655 wait complete. 2023-11-25T05:26:21.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.406+0000 7f33ce0e0640 10 monclient: build_initial_monmap 2023-11-25T05:26:21.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.406+0000 7f33ce0e0640 10 monclient: monmap: 2023-11-25T05:26:21.413 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:21.413 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:21.413 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:21.413 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:21.413 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:21.413 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:21.413 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:21.413 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:21.414 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:21.414 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:21.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.407+0000 7f33ce0e0640 1 librados: starting msgr at 2023-11-25T05:26:21.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.407+0000 7f33ce0e0640 1 librados: starting objecter 2023-11-25T05:26:21.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.407+0000 7f33ce0e0640 1 Processor -- start 2023-11-25T05:26:21.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.407+0000 7f33ce0e0640 1 -- start start 2023-11-25T05:26:21.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.407+0000 7f33ce0e0640 1 librados: setting wanted keys 2023-11-25T05:26:21.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.407+0000 7f33ce0e0640 1 librados: calling monclient init 2023-11-25T05:26:21.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.407+0000 7f33ce0e0640 10 monclient: init 2023-11-25T05:26:21.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:21.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-25T05:26:21.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33c80a8910 0x7f33c80b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:21.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 10 monclient(hunting): picked mon.noname-b con 0x7f33c80a8910 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:21.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33c8141f90 0x7f33c813b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:21.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 10 monclient(hunting): picked mon.noname-a con 0x7f33c8141f90 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:21.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33c813a8d0 0x7f33c813b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:21.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 10 monclient(hunting): picked mon.noname-c con 0x7f33c813a8d0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:21.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:21.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c813aef0 con 0x7f33c8141f90 2023-11-25T05:26:21.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:21.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c80ac180 con 0x7f33c80a8910 2023-11-25T05:26:21.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:21.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c813bd60 con 0x7f33c813a8d0 2023-11-25T05:26:21.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:21.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.408+0000 7f33ce0e0640 10 monclient(hunting): authenticate will time out at 2707.651015s 2023-11-25T05:26:21.416 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:03.607 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:21.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.410+0000 7f33c6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33c8141f90 0x7f33c813b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:21.416 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:03.611 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:21.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.411+0000 7f33c6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33c8141f90 0x7f33c813b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43726/0 (socket says 172.21.15.7:43726) 2023-11-25T05:26:21.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.411+0000 7f33c6ffd640 1 -- 172.21.15.7:0/577242153 learned_addr learned my addr 172.21.15.7:0/577242153 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:21.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.411+0000 7f33c6ffd640 10 monclient(hunting): get_auth_request con 0x7f33c8141f90 auth_method 0 2023-11-25T05:26:21.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.411+0000 7f33c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:21.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.411+0000 7f33c6ffd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:21.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.411+0000 7f33c6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:21.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.413+0000 7f33c77fe640 1 --2- 172.21.15.7:0/577242153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33c80a8910 0x7f33c80b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:21.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.414+0000 7f33c77fe640 10 monclient(hunting): get_auth_request con 0x7f33c80a8910 auth_method 0 2023-11-25T05:26:21.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.414+0000 7f33c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:21.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.414+0000 7f33c77fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:21.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.414+0000 7f33c77fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:21.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.415+0000 7f33c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:21.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.415+0000 7f33c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:21.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.415+0000 7f33c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:21.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.418+0000 7f33c77fe640 10 monclient(hunting): handle_auth_done global_id 21250 payload 824 2023-11-25T05:26:21.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.418+0000 7f33c77fe640 1 -- 172.21.15.7:0/577242153 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33c813a8d0 msgr2=0x7f33c813b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:21.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.418+0000 7f33c77fe640 1 --2- 172.21.15.7:0/577242153 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33c813a8d0 0x7f33c813b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.418+0000 7f33c77fe640 1 -- 172.21.15.7:0/577242153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33c8141f90 msgr2=0x7f33c813b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:21.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.418+0000 7f33c77fe640 1 --2- 172.21.15.7:0/577242153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33c8141f90 0x7f33c813b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.418+0000 7f33c77fe640 10 monclient: _finish_hunting 0 2023-11-25T05:26:21.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.418+0000 7f33c77fe640 1 monclient: found mon.noname-b 2023-11-25T05:26:21.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.418+0000 7f33c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:21.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.418+0000 7f33c77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:26:21.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.418+0000 7f33c77fe640 1 -- 172.21.15.7:0/577242153 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c0009b30 con 0x7f33c80a8910 2023-11-25T05:26:21.441 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:03.639 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:21.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.483+0000 7f33c77fe640 1 --2- 172.21.15.7:0/577242153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33c80a8910 0x7f33c80b6a90 secure :-1 s=READY pgs=5155 cs=0 l=1 rev1=1 crypto rx=0x7f33b800b3f0 tx=0x7f33b800b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:21.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.483+0000 7f33affff640 10 client.?.objecter ms_handle_connect 0x7f33c80a8910 2023-11-25T05:26:21.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.483+0000 7f33affff640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:21.485 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:03.629 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:21.489 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:03.688 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33affff640 1 -- 172.21.15.7:0/577242153 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b8011830 con 0x7f33c80a8910 2023-11-25T05:26:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33affff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33affff640 10 monclient: dump: 2023-11-25T05:26:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:21.496 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33affff640 10 monclient: _finish_auth 0 2023-11-25T05:26:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33affff640 10 monclient: _check_auth_tickets 2023-11-25T05:26:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33affff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33affff640 1 -- 172.21.15.7:0/577242153 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b8011990 con 0x7f33c80a8910 2023-11-25T05:26:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33affff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33affff640 1 -- 172.21.15.7:0/577242153 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b8011d40 con 0x7f33c80a8910 2023-11-25T05:26:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33affff640 10 monclient: dump: 2023-11-25T05:26:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:21.498 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:21.498 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:21.498 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:21.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33ce0e0640 5 monclient: authenticate success, global_id 21250 2023-11-25T05:26:21.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33ce0e0640 10 monclient: _renew_subs 2023-11-25T05:26:21.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33ce0e0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:21.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33c5ffb640 4 set_mon_vals no callback set 2023-11-25T05:26:21.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33ce0e0640 1 -- 172.21.15.7:0/577242153 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c814c160 con 0x7f33c80a8910 2023-11-25T05:26:21.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33ce0e0640 10 client.21250.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:21.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33ce0e0640 10 monclient: _renew_subs 2023-11-25T05:26:21.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33ce0e0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:21.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33ce0e0640 1 -- 172.21.15.7:0/577242153 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c81da690 con 0x7f33c80a8910 2023-11-25T05:26:21.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.490+0000 7f33ce0e0640 1 librados: init done 2023-11-25T05:26:21.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.491+0000 7f33ce0e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:21.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.491+0000 7f33ce0e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:21.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.491+0000 7f33ce0e0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:21.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.491+0000 7f33ce0e0640 1 -- 172.21.15.7:0/577242153 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c8075b60 con 0x7f33c80a8910 2023-11-25T05:26:21.499 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:03.693 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:21.500 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:03.636 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:21.525 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:03.724 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:21.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.750+0000 7f33affff640 1 -- 172.21.15.7:0/577242153 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f33b801a400 con 0x7f33c80a8910 2023-11-25T05:26:21.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.750+0000 7f33affff640 1 --2- 172.21.15.7:0/577242153 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f33b406c820 0x7f33b406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:21.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.751+0000 7f33c6ffd640 1 --2- 172.21.15.7:0/577242153 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f33b406c820 0x7f33b406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:21.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.751+0000 7f33c6ffd640 10 monclient: get_auth_request con 0x7f33b406c820 auth_method 0 2023-11-25T05:26:21.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.751+0000 7f33affff640 10 client.21250.objecter ms_handle_connect 0x7f33b406c820 2023-11-25T05:26:21.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.751+0000 7f33c6ffd640 1 --2- 172.21.15.7:0/577242153 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f33b406c820 0x7f33b406ece0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f33c8133ea0 tx=0x7f33bc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:21.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.754+0000 7f33affff640 1 -- 172.21.15.7:0/577242153 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f33b8059150 con 0x7f33c80a8910 2023-11-25T05:26:21.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.754+0000 7f33affff640 10 client.21250.objecter ms_dispatch 0x7f33c8000b90 osd_map(918..918 src has 501..918) v4 2023-11-25T05:26:21.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.754+0000 7f33affff640 3 client.21250.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:21.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.754+0000 7f33affff640 3 client.21250.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:21.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.754+0000 7f33affff640 20 client.21250.objecter dump_active .. 0 homeless 2023-11-25T05:26:21.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.765+0000 7f33affff640 1 -- 172.21.15.7:0/577242153 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f33b8054d90 con 0x7f33c80a8910 2023-11-25T05:26:21.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.765+0000 7f33affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:21.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.765+0000 7f33affff640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:26:21.821 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:03.965 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.984+0000 7f33ce0e0640 1 -- 172.21.15.7:0/577242153 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f33c813e320 con 0x7f33b406c820 2023-11-25T05:26:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.985+0000 7f33affff640 1 -- 172.21.15.7:0/577242153 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f33c813e320 con 0x7f33b406c820 2023-11-25T05:26:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 1 to deep-scrub 2023-11-25T05:26:21.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 10 librados: watch_flush enter 2023-11-25T05:26:21.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 10 librados: watch_flush exit 2023-11-25T05:26:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 20 client.21250.objecter shutdown clearing up homeless session... 2023-11-25T05:26:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 10 client.21250.objecter successfully canceled tick 2023-11-25T05:26:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 1 -- 172.21.15.7:0/577242153 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f33b406c820 msgr2=0x7f33b406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 1 --2- 172.21.15.7:0/577242153 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f33b406c820 0x7f33b406ece0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f33c8133ea0 tx=0x7f33bc009040 comp rx=0 tx=0).stop 2023-11-25T05:26:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 10 monclient: shutdown 2023-11-25T05:26:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 1 -- 172.21.15.7:0/577242153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33c80a8910 msgr2=0x7f33c80b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 1 --2- 172.21.15.7:0/577242153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33c80a8910 0x7f33c80b6a90 secure :-1 s=READY pgs=5155 cs=0 l=1 rev1=1 crypto rx=0x7f33b800b3f0 tx=0x7f33b800b8b0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 1 -- 172.21.15.7:0/577242153 shutdown_connections 2023-11-25T05:26:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 1 --2- 172.21.15.7:0/577242153 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f33b406c820 0x7f33b406ece0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 1 --2- 172.21.15.7:0/577242153 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33c813a8d0 0x7f33c813b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 1 --2- 172.21.15.7:0/577242153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33c8141f90 0x7f33c813b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 1 --2- 172.21.15.7:0/577242153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33c80a8910 0x7f33c80b6a90 unknown :-1 s=CLOSED pgs=5155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:21.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 1 -- 172.21.15.7:0/577242153 >> 172.21.15.7:0/577242153 conn(0x7f33c80b7780 msgr2=0x7f33c80b7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:21.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 1 -- 172.21.15.7:0/577242153 shutdown_connections 2023-11-25T05:26:21.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 1 -- 172.21.15.7:0/577242153 wait complete. 2023-11-25T05:26:21.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:21.987+0000 7f33ce0e0640 1 librados: shutdown 2023-11-25T05:26:22.005 INFO:tasks.ceph:Scrubbing osd.2 2023-11-25T05:26:22.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-11-25T05:26:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.120+0000 7f002b683640 10 monclient: get_monmap_and_config 2023-11-25T05:26:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.120+0000 7f002b683640 10 monclient: build_initial_monmap 2023-11-25T05:26:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.120+0000 7f002b683640 10 monclient: monmap: 2023-11-25T05:26:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:22.120970+0000 2023-11-25T05:26:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:22.120970+0000 2023-11-25T05:26:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:22.122 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.120+0000 7f002b683640 1 Processor -- start 2023-11-25T05:26:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.120+0000 7f002b683640 1 -- start start 2023-11-25T05:26:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.120+0000 7f002b683640 10 monclient: init 2023-11-25T05:26:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.121+0000 7f002b683640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.121+0000 7f002b683640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-11-25T05:26:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.121+0000 7f002b683640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00240a88a0 0x7f00240a8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.121+0000 7f002b683640 10 monclient(hunting): picked mon.noname-a con 0x7f00240a88a0 addr v2:172.21.15.7:3300/0 2023-11-25T05:26:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.121+0000 7f002b683640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f002413f0b0 0x7f002413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.121+0000 7f002b683640 10 monclient(hunting): picked mon.noname-b con 0x7f002413f0b0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.121+0000 7f002b683640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0024145480 addr v1:172.21.15.7:6789/0 2023-11-25T05:26:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.121+0000 7f002b683640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00240b17d0 con 0x7f0024145480 2023-11-25T05:26:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.121+0000 7f002b683640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.121+0000 7f002b683640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00240aac60 con 0x7f00240a88a0 2023-11-25T05:26:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.121+0000 7f002b683640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.121+0000 7f002b683640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00240b1560 con 0x7f002413f0b0 2023-11-25T05:26:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.121+0000 7f002b683640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.121+0000 7f002b683640 10 monclient(hunting): authenticate will time out at 2708.364475s 2023-11-25T05:26:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.125+0000 7f002941f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00240a88a0 0x7f00240a8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:22.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.126+0000 7f0028c1e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f002413f0b0 0x7f002413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:22.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.128+0000 7f002941f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00240a88a0 0x7f00240a8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43740/0 (socket says 172.21.15.7:43740) 2023-11-25T05:26:22.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.128+0000 7f002941f640 1 -- 172.21.15.7:0/2581781103 learned_addr learned my addr 172.21.15.7:0/2581781103 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:22.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.128+0000 7f002941f640 10 monclient(hunting): get_auth_request con 0x7f00240a88a0 auth_method 0 2023-11-25T05:26:22.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.128+0000 7f002941f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:22.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.128+0000 7f002941f640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:22.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.128+0000 7f002941f640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:22.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.128+0000 7f0028c1e640 10 monclient(hunting): get_auth_request con 0x7f002413f0b0 auth_method 0 2023-11-25T05:26:22.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.128+0000 7f0028c1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:22.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.128+0000 7f0028c1e640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:22.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.128+0000 7f0028c1e640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:22.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.128+0000 7f002941f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:22.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.128+0000 7f002941f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:22.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.128+0000 7f002941f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:22.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.129+0000 7f0028c1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:22.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.129+0000 7f0028c1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:22.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.129+0000 7f0028c1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:22.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.133+0000 7f002941f640 10 monclient(hunting): handle_auth_done global_id 15918 payload 274 2023-11-25T05:26:22.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.133+0000 7f002941f640 1 -- 172.21.15.7:0/2581781103 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f002413f0b0 msgr2=0x7f002413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:22.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.133+0000 7f002941f640 1 --2- 172.21.15.7:0/2581781103 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f002413f0b0 0x7f002413f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.133+0000 7f002941f640 1 -- 172.21.15.7:0/2581781103 >> v1:172.21.15.7:6789/0 conn(0x7f0024145480 legacy=0x7f0024145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:22.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.133+0000 7f002941f640 10 monclient: _finish_hunting 0 2023-11-25T05:26:22.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.133+0000 7f002941f640 1 monclient: found mon.noname-a 2023-11-25T05:26:22.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.133+0000 7f002941f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:22.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.133+0000 7f002941f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:26:22.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.133+0000 7f002941f640 1 -- 172.21.15.7:0/2581781103 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002413fa10 con 0x7f00240a88a0 2023-11-25T05:26:22.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.140+0000 7f002941f640 1 --2- 172.21.15.7:0/2581781103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00240a88a0 0x7f00240a8c80 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7f00200098a0 tx=0x7f002002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd253f17dc10e7db server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:22.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f001bfff640 1 -- 172.21.15.7:0/2581781103 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00200317c0 con 0x7f00240a88a0 2023-11-25T05:26:22.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f001bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:22.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f001bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:26:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f001bfff640 10 monclient: dump: 2023-11-25T05:26:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:22.158 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f001bfff640 10 monclient: _finish_auth 0 2023-11-25T05:26:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f001bfff640 10 monclient: _check_auth_tickets 2023-11-25T05:26:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f001bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f001bfff640 1 -- 172.21.15.7:0/2581781103 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0020031920 con 0x7f00240a88a0 2023-11-25T05:26:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f001bfff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f001bfff640 1 -- 172.21.15.7:0/2581781103 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0020031cb0 con 0x7f00240a88a0 2023-11-25T05:26:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f001bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f001bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:26:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f001bfff640 10 monclient: dump: 2023-11-25T05:26:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.160 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:22.160 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:22.160 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:22.160 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:22.160 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:22.160 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:22.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f002b683640 5 monclient: authenticate success, global_id 15918 2023-11-25T05:26:22.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f002b683640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:22.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.156+0000 7f002b683640 4 set_mon_vals no callback set 2023-11-25T05:26:22.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.157+0000 7f002b683640 10 monclient: shutdown 2023-11-25T05:26:22.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.157+0000 7f002b683640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:22.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.157+0000 7f002b683640 1 -- 172.21.15.7:0/2581781103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00240a88a0 msgr2=0x7f00240a8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:22.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.157+0000 7f002b683640 1 --2- 172.21.15.7:0/2581781103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00240a88a0 0x7f00240a8c80 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7f00200098a0 tx=0x7f002002b860 comp rx=0 tx=0).stop 2023-11-25T05:26:22.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.157+0000 7f002b683640 1 -- 172.21.15.7:0/2581781103 shutdown_connections 2023-11-25T05:26:22.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.157+0000 7f002b683640 1 --2- 172.21.15.7:0/2581781103 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f002413f0b0 0x7f002413f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.157+0000 7f002b683640 1 --2- 172.21.15.7:0/2581781103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00240a88a0 0x7f00240a8c80 unknown :-1 s=CLOSED pgs=3147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.157+0000 7f002b683640 1 -- 172.21.15.7:0/2581781103 >> 172.21.15.7:0/2581781103 conn(0x7f00240619a0 msgr2=0x7f00240b7780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:22.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.157+0000 7f002b683640 1 -- 172.21.15.7:0/2581781103 shutdown_connections 2023-11-25T05:26:22.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.157+0000 7f002b683640 1 -- 172.21.15.7:0/2581781103 wait complete. 2023-11-25T05:26:22.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.157+0000 7f002b683640 10 monclient: build_initial_monmap 2023-11-25T05:26:22.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.157+0000 7f002b683640 10 monclient: monmap: 2023-11-25T05:26:22.163 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:22.163 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:22.163 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:22.163 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:22.163 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:22.163 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:22.163 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:22.163 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:22.163 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:22.164 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:22.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.158+0000 7f002b683640 1 librados: starting msgr at 2023-11-25T05:26:22.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.158+0000 7f002b683640 1 librados: starting objecter 2023-11-25T05:26:22.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.158+0000 7f002b683640 1 Processor -- start 2023-11-25T05:26:22.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.158+0000 7f002b683640 1 -- start start 2023-11-25T05:26:22.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.158+0000 7f002b683640 1 librados: setting wanted keys 2023-11-25T05:26:22.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.158+0000 7f002b683640 1 librados: calling monclient init 2023-11-25T05:26:22.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.158+0000 7f002b683640 10 monclient: init 2023-11-25T05:26:22.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.158+0000 7f002b683640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:22.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.158+0000 7f002b683640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-25T05:26:22.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.158+0000 7f002b683640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00240a88a0 0x7f0024137e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:22.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.158+0000 7f002b683640 10 monclient(hunting): picked mon.noname-c con 0x7f00240a88a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:22.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.158+0000 7f002b683640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f002413f0b0 0x7f0024138380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:22.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.159+0000 7f002b683640 10 monclient(hunting): picked mon.noname-a con 0x7f002413f0b0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:22.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.159+0000 7f002b683640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0024145480 0x7f00241338f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:22.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.159+0000 7f002b683640 10 monclient(hunting): picked mon.noname-b con 0x7f0024145480 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:22.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.159+0000 7f002b683640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:22.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.159+0000 7f002b683640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00241d6ed0 con 0x7f002413f0b0 2023-11-25T05:26:22.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.159+0000 7f002b683640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:22.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.159+0000 7f002b683640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002413fbd0 con 0x7f0024145480 2023-11-25T05:26:22.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.159+0000 7f002b683640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:22.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.159+0000 7f002b683640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0024141780 con 0x7f00240a88a0 2023-11-25T05:26:22.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.159+0000 7f002b683640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:22.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.159+0000 7f002b683640 10 monclient(hunting): authenticate will time out at 2708.401880s 2023-11-25T05:26:22.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.161+0000 7f002941f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00240a88a0 0x7f0024137e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:22.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.162+0000 7f0028c1e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f002413f0b0 0x7f0024138380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.162+0000 7f002941f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00240a88a0 0x7f0024137e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38292/0 (socket says 172.21.15.7:38292) 2023-11-25T05:26:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.162+0000 7f002941f640 1 -- 172.21.15.7:0/1365222648 learned_addr learned my addr 172.21.15.7:0/1365222648 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.162+0000 7f002941f640 10 monclient(hunting): get_auth_request con 0x7f00240a88a0 auth_method 0 2023-11-25T05:26:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.162+0000 7f002941f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.162+0000 7f002941f640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.162+0000 7f002941f640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.163+0000 7f0028c1e640 10 monclient(hunting): get_auth_request con 0x7f002413f0b0 auth_method 0 2023-11-25T05:26:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.163+0000 7f0028c1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.163+0000 7f0028c1e640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.163+0000 7f0028c1e640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.164+0000 7f002941f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.164+0000 7f002941f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.164+0000 7f002941f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:22.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.165+0000 7f0028c1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:22.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.165+0000 7f0028c1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:22.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.165+0000 7f0028c1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:22.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.166+0000 7f0029c20640 1 --2- 172.21.15.7:0/1365222648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0024145480 0x7f00241338f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:22.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.167+0000 7f002941f640 10 monclient(hunting): handle_auth_done global_id 17399 payload 824 2023-11-25T05:26:22.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.167+0000 7f002941f640 1 -- 172.21.15.7:0/1365222648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0024145480 msgr2=0x7f00241338f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:22.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.167+0000 7f002941f640 1 --2- 172.21.15.7:0/1365222648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0024145480 0x7f00241338f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.167+0000 7f002941f640 1 -- 172.21.15.7:0/1365222648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f002413f0b0 msgr2=0x7f0024138380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:22.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.167+0000 7f002941f640 1 --2- 172.21.15.7:0/1365222648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f002413f0b0 0x7f0024138380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.167+0000 7f002941f640 10 monclient: _finish_hunting 0 2023-11-25T05:26:22.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.167+0000 7f002941f640 1 monclient: found mon.noname-c 2023-11-25T05:26:22.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.167+0000 7f002941f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:22.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.167+0000 7f002941f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:26:22.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.167+0000 7f002941f640 1 -- 172.21.15.7:0/1365222648 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00200094d0 con 0x7f00240a88a0 2023-11-25T05:26:22.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.174+0000 7f002941f640 1 --2- 172.21.15.7:0/1365222648 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00240a88a0 0x7f0024137e40 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f0020000c00 tx=0x7f00200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:22.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.174+0000 7f0019ffb640 10 client.?.objecter ms_handle_connect 0x7f00240a88a0 2023-11-25T05:26:22.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.174+0000 7f0019ffb640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:22.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f0019ffb640 1 -- 172.21.15.7:0/1365222648 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0020032860 con 0x7f00240a88a0 2023-11-25T05:26:22.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f0019ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:22.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f0019ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:26:22.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f0019ffb640 10 monclient: dump: 2023-11-25T05:26:22.189 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:22.189 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:22.190 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.190 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.190 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:22.190 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:22.190 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:22.191 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:22.191 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:22.191 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:22.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f0019ffb640 10 monclient: _finish_auth 0 2023-11-25T05:26:22.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f0019ffb640 10 monclient: _check_auth_tickets 2023-11-25T05:26:22.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f0019ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:22.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f0019ffb640 1 -- 172.21.15.7:0/1365222648 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00200329c0 con 0x7f00240a88a0 2023-11-25T05:26:22.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f002b683640 5 monclient: authenticate success, global_id 17399 2023-11-25T05:26:22.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f002b683640 10 monclient: _renew_subs 2023-11-25T05:26:22.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f002b683640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:26:22.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f002b683640 1 -- 172.21.15.7:0/1365222648 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0024142350 con 0x7f00240a88a0 2023-11-25T05:26:22.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f0019ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:22.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f0019ffb640 1 -- 172.21.15.7:0/1365222648 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0020032d50 con 0x7f00240a88a0 2023-11-25T05:26:22.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f0019ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:22.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f0019ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:26:22.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f0019ffb640 10 monclient: dump: 2023-11-25T05:26:22.197 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:22.197 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:22.197 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.197 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.197 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:22.197 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:22.197 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:22.197 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:22.197 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:22.197 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:22.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f001b7fe640 4 set_mon_vals no callback set 2023-11-25T05:26:22.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f002b683640 10 client.17399.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:22.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f002b683640 10 monclient: _renew_subs 2023-11-25T05:26:22.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f002b683640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:26:22.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f002b683640 1 -- 172.21.15.7:0/1365222648 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0024133e30 con 0x7f00240a88a0 2023-11-25T05:26:22.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.187+0000 7f002b683640 1 librados: init done 2023-11-25T05:26:22.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.188+0000 7f002b683640 10 client.17399.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:22.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.188+0000 7f002b683640 15 client.17399.objecter _session_command_op_assign -1 1 2023-11-25T05:26:22.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.188+0000 7f002b683640 20 client.17399.objecter _get_session osd=-1 returning homeless 2023-11-25T05:26:22.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.188+0000 7f002b683640 10 client.17399.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:22.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.188+0000 7f002b683640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:26:22.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.188+0000 7f002b683640 1 -- 172.21.15.7:0/1365222648 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0024142760 con 0x7f00240a88a0 2023-11-25T05:26:22.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.217+0000 7f0019ffb640 1 -- 172.21.15.7:0/1365222648 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f002004e450 con 0x7f00240a88a0 2023-11-25T05:26:22.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.217+0000 7f0019ffb640 1 --2- 172.21.15.7:0/1365222648 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f000806c7d0 0x7f000806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:22.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.218+0000 7f0028c1e640 1 --2- 172.21.15.7:0/1365222648 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f000806c7d0 0x7f000806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:22.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.218+0000 7f0028c1e640 10 monclient: get_auth_request con 0x7f000806c7d0 auth_method 0 2023-11-25T05:26:22.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.218+0000 7f0028c1e640 1 --2- 172.21.15.7:0/1365222648 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f000806c7d0 0x7f000806ec90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0024137390 tx=0x7f0014009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:22.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.218+0000 7f0019ffb640 10 client.17399.objecter ms_handle_connect 0x7f000806c7d0 2023-11-25T05:26:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 1 -- 172.21.15.7:0/1365222648 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f002007b130 con 0x7f00240a88a0 2023-11-25T05:26:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 10 client.17399.objecter ms_dispatch 0x7f0024000b90 osd_map(918..918 src has 501..918) v4 2023-11-25T05:26:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 3 client.17399.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 3 client.17399.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 10 client.17399.objecter checking command 1 2023-11-25T05:26:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 1 --2- 172.21.15.7:0/1365222648 >> [v2:172.21.15.7:6808/1372486177,v1:172.21.15.7:6809/1372486177] conn(0x7f0008074050 0x7f00080764f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 20 client.17399.objecter _get_session s=0x7f0008073980 osd=2 3 2023-11-25T05:26:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 20 client.17399.objecter put_session s=0x7f0008073980 osd=2 3 2023-11-25T05:26:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 15 client.17399.objecter _session_command_op_remove -1 1 2023-11-25T05:26:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 20 client.17399.objecter _get_session s=0x7f0008073980 osd=2 3 2023-11-25T05:26:22.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 20 client.17399.objecter get_session s=0x7f0008073980 osd=2 3 2023-11-25T05:26:22.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 15 client.17399.objecter _session_command_op_assign 2 1 2023-11-25T05:26:22.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 20 client.17399.objecter put_session s=0x7f0008073980 osd=2 4 2023-11-25T05:26:22.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 10 client.17399.objecter _send_command 1 2023-11-25T05:26:22.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 1 -- 172.21.15.7:0/1365222648 --> [v2:172.21.15.7:6808/1372486177,v1:172.21.15.7:6809/1372486177] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0008076bc0 con 0x7f0008074050 2023-11-25T05:26:22.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 20 client.17399.objecter dump_active .. 0 homeless 2023-11-25T05:26:22.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 1 -- 172.21.15.7:0/1365222648 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_get_version_reply(handle=1 version=918) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0020076d70 con 0x7f00240a88a0 2023-11-25T05:26:22.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.226+0000 7f0019ffb640 10 monclient: handle_get_version_reply finishing 1 version 918 2023-11-25T05:26:22.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.227+0000 7f0029c20640 1 --2- 172.21.15.7:0/1365222648 >> [v2:172.21.15.7:6808/1372486177,v1:172.21.15.7:6809/1372486177] conn(0x7f0008074050 0x7f00080764f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:22.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.227+0000 7f0029c20640 10 monclient: get_auth_request con 0x7f0008074050 auth_method 0 2023-11-25T05:26:22.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.230+0000 7f0019ffb640 10 client.17399.objecter ms_handle_connect 0x7f0008074050 2023-11-25T05:26:22.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.230+0000 7f0029c20640 1 --2- 172.21.15.7:0/1365222648 >> [v2:172.21.15.7:6808/1372486177,v1:172.21.15.7:6809/1372486177] conn(0x7f0008074050 0x7f00080764f0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.254+0000 7f0019ffb640 1 -- 172.21.15.7:0/1365222648 <== osd.2 v2:172.21.15.7:6808/1372486177 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f0008076bc0 con 0x7f0008074050 2023-11-25T05:26:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.254+0000 7f0019ffb640 10 client.17399.objecter ms_dispatch 0x7f0024000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:26:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.254+0000 7f0019ffb640 10 client.17399.objecter _finish_command 1 = system:0 2023-11-25T05:26:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.255+0000 7f0019ffb640 20 client.17399.objecter put_session s=0x7f0008073980 osd=2 4 2023-11-25T05:26:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.255+0000 7f0019ffb640 15 client.17399.objecter _session_command_op_remove 2 1 2023-11-25T05:26:22.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.268+0000 7f002b683640 10 client.17399.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-11-25T05:26:22.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.268+0000 7f002b683640 15 client.17399.objecter _session_command_op_assign -1 2 2023-11-25T05:26:22.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.268+0000 7f002b683640 20 client.17399.objecter _get_session s=0x7f0008073980 osd=2 3 2023-11-25T05:26:22.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.268+0000 7f002b683640 20 client.17399.objecter put_session s=0x7f0008073980 osd=2 3 2023-11-25T05:26:22.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.268+0000 7f002b683640 20 client.17399.objecter _get_session s=0x7f0008073980 osd=2 3 2023-11-25T05:26:22.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.268+0000 7f002b683640 15 client.17399.objecter _session_command_op_remove -1 2 2023-11-25T05:26:22.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.268+0000 7f002b683640 20 client.17399.objecter get_session s=0x7f0008073980 osd=2 3 2023-11-25T05:26:22.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.268+0000 7f002b683640 15 client.17399.objecter _session_command_op_assign 2 2 2023-11-25T05:26:22.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.268+0000 7f002b683640 20 client.17399.objecter put_session s=0x7f0008073980 osd=2 4 2023-11-25T05:26:22.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.268+0000 7f002b683640 10 client.17399.objecter _send_command 2 2023-11-25T05:26:22.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.268+0000 7f002b683640 1 -- 172.21.15.7:0/1365222648 --> [v2:172.21.15.7:6808/1372486177,v1:172.21.15.7:6809/1372486177] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f002414c660 con 0x7f0008074050 2023-11-25T05:26:22.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.288+0000 7f0019ffb640 1 -- 172.21.15.7:0/1365222648 <== osd.2 v2:172.21.15.7:6808/1372486177 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f001c007cd0 con 0x7f0008074050 2023-11-25T05:26:22.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.288+0000 7f0019ffb640 10 client.17399.objecter ms_dispatch 0x7f0024000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:26:22.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.288+0000 7f0019ffb640 10 client.17399.objecter _finish_command 2 = system:0 2023-11-25T05:26:22.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.288+0000 7f0019ffb640 20 client.17399.objecter put_session s=0x7f0008073980 osd=2 4 2023-11-25T05:26:22.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.288+0000 7f0019ffb640 15 client.17399.objecter _session_command_op_remove 2 2 2023-11-25T05:26:22.291 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-11-25T05:26:22.291 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-25T05:26:22.291 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-11-25T05:26:22.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 10 librados: watch_flush enter 2023-11-25T05:26:22.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 10 librados: watch_flush exit 2023-11-25T05:26:22.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 10 client.17399.objecter close_session for osd.2 2023-11-25T05:26:22.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 1 -- 172.21.15.7:0/1365222648 >> [v2:172.21.15.7:6808/1372486177,v1:172.21.15.7:6809/1372486177] conn(0x7f0008074050 msgr2=0x7f00080764f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:22.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 1 --2- 172.21.15.7:0/1365222648 >> [v2:172.21.15.7:6808/1372486177,v1:172.21.15.7:6809/1372486177] conn(0x7f0008074050 0x7f00080764f0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 20 client.17399.objecter put_session s=0x7f0008073980 osd=2 1 2023-11-25T05:26:22.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 20 client.17399.objecter shutdown clearing up homeless session... 2023-11-25T05:26:22.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 10 client.17399.objecter successfully canceled tick 2023-11-25T05:26:22.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 1 -- 172.21.15.7:0/1365222648 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f000806c7d0 msgr2=0x7f000806ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:22.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 1 --2- 172.21.15.7:0/1365222648 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f000806c7d0 0x7f000806ec90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0024137390 tx=0x7f0014009190 comp rx=0 tx=0).stop 2023-11-25T05:26:22.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 10 monclient: shutdown 2023-11-25T05:26:22.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:22.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 1 -- 172.21.15.7:0/1365222648 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00240a88a0 msgr2=0x7f0024137e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:22.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 1 --2- 172.21.15.7:0/1365222648 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00240a88a0 0x7f0024137e40 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f0020000c00 tx=0x7f00200046c0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f0029c20640 1 -- 172.21.15.7:0/1365222648 reap_dead start 2023-11-25T05:26:22.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 1 -- 172.21.15.7:0/1365222648 shutdown_connections 2023-11-25T05:26:22.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 1 -- 172.21.15.7:0/1365222648 >> 172.21.15.7:0/1365222648 conn(0x7f00240619a0 msgr2=0x7f00240b2080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:22.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 1 -- 172.21.15.7:0/1365222648 shutdown_connections 2023-11-25T05:26:22.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 1 -- 172.21.15.7:0/1365222648 wait complete. 2023-11-25T05:26:22.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.289+0000 7f002b683640 1 librados: shutdown 2023-11-25T05:26:22.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-11-25T05:26:22.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.410+0000 7f3db6a21640 10 monclient: get_monmap_and_config 2023-11-25T05:26:22.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.410+0000 7f3db6a21640 10 monclient: build_initial_monmap 2023-11-25T05:26:22.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.411+0000 7f3db6a21640 10 monclient: monmap: 2023-11-25T05:26:22.412 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:22.412 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:22.412 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:22.411920+0000 2023-11-25T05:26:22.412 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:22.411920+0000 2023-11-25T05:26:22.413 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:22.413 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:22.413 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:22.413 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:22.413 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:22.413 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:22.413 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:22.413 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.411+0000 7f3db6a21640 1 Processor -- start 2023-11-25T05:26:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.411+0000 7f3db6a21640 1 -- start start 2023-11-25T05:26:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.411+0000 7f3db6a21640 10 monclient: init 2023-11-25T05:26:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.412+0000 7f3db6a21640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.412+0000 7f3db6a21640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-11-25T05:26:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.412+0000 7f3db6a21640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3db00a8910 0x7f3db013ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.412+0000 7f3db6a21640 10 monclient(hunting): picked mon.noname-b con 0x7f3db00a8910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.412+0000 7f3db6a21640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3db00a9b70 addr v1:172.21.15.7:6789/0 2023-11-25T05:26:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.412+0000 7f3db6a21640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3db0141f90 addr v1:172.21.15.90:6789/0 2023-11-25T05:26:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.412+0000 7f3db6a21640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db00a8cf0 con 0x7f3db00a9b70 2023-11-25T05:26:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.412+0000 7f3db6a21640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db00aabd0 con 0x7f3db0141f90 2023-11-25T05:26:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.412+0000 7f3db6a21640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.412+0000 7f3db6a21640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db00aaeb0 con 0x7f3db00a8910 2023-11-25T05:26:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.412+0000 7f3db6a21640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.412+0000 7f3db6a21640 10 monclient(hunting): authenticate will time out at 2708.655358s 2023-11-25T05:26:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.413+0000 7f3daffff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3db00a8910 0x7f3db013ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.413+0000 7f3daf7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3db00a9b70 0x7f3db013f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49908/0 (socket says 172.21.15.7:49908) 2023-11-25T05:26:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.413+0000 7f3daf7fe640 1 -- 172.21.15.7:0/361000789 learned_addr learned my addr 172.21.15.7:0/361000789 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:26:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.413+0000 7f3daffff640 10 monclient(hunting): get_auth_request con 0x7f3db00a8910 auth_method 0 2023-11-25T05:26:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.413+0000 7f3daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.413+0000 7f3daffff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:22.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.413+0000 7f3daffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:22.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.414+0000 7f3daffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:22.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.414+0000 7f3daffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:22.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.414+0000 7f3daffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:22.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.416+0000 7f3daffff640 10 monclient(hunting): handle_auth_done global_id 17402 payload 274 2023-11-25T05:26:22.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.416+0000 7f3daffff640 1 -- 172.21.15.7:0/361000789 >> v1:172.21.15.90:6789/0 conn(0x7f3db0141f90 legacy=0x7f3db01423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.416+0000 7f3daffff640 1 -- 172.21.15.7:0/361000789 >> v1:172.21.15.7:6789/0 conn(0x7f3db00a9b70 legacy=0x7f3db013f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.416+0000 7f3daffff640 10 monclient: _finish_hunting 0 2023-11-25T05:26:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.416+0000 7f3daffff640 1 monclient: found mon.noname-b 2023-11-25T05:26:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.416+0000 7f3daffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.416+0000 7f3daffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:26:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.416+0000 7f3daffff640 1 -- 172.21.15.7:0/361000789 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db00a9720 con 0x7f3db00a8910 2023-11-25T05:26:22.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.418+0000 7f3daffff640 1 --2- 172.21.15.7:0/361000789 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3db00a8910 0x7f3db013ec60 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f3da00097d0 tx=0x7f3da002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db2e816e27b066c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3daeffd640 1 -- 172.21.15.7:0/361000789 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da0031800 con 0x7f3db00a8910 2023-11-25T05:26:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3daeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3daeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3daeffd640 10 monclient: dump: 2023-11-25T05:26:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:22.425 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3daeffd640 10 monclient: _finish_auth 0 2023-11-25T05:26:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3daeffd640 10 monclient: _check_auth_tickets 2023-11-25T05:26:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3daeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3daeffd640 1 -- 172.21.15.7:0/361000789 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da0031960 con 0x7f3db00a8910 2023-11-25T05:26:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3db6a21640 5 monclient: authenticate success, global_id 17402 2023-11-25T05:26:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3db6a21640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:26:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3daeffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3db6a21640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3daeffd640 1 -- 172.21.15.7:0/361000789 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da0031cf0 con 0x7f3db00a8910 2023-11-25T05:26:22.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3db6a21640 4 set_mon_vals no callback set 2023-11-25T05:26:22.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3daeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:22.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3daeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:26:22.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.422+0000 7f3daeffd640 10 monclient: dump: 2023-11-25T05:26:22.426 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:22.426 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:22.426 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.426 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.426 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:22.426 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:22.426 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:22.426 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:22.426 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:22.427 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:22.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.423+0000 7f3db6a21640 10 monclient: shutdown 2023-11-25T05:26:22.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.423+0000 7f3db6a21640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:22.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.423+0000 7f3db6a21640 1 -- 172.21.15.7:0/361000789 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3db00a8910 msgr2=0x7f3db013ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:22.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.423+0000 7f3db6a21640 1 --2- 172.21.15.7:0/361000789 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3db00a8910 0x7f3db013ec60 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f3da00097d0 tx=0x7f3da002b8a0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.423+0000 7f3db6a21640 1 -- 172.21.15.7:0/361000789 shutdown_connections 2023-11-25T05:26:22.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.423+0000 7f3db6a21640 1 --2- 172.21.15.7:0/361000789 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3db00a8910 0x7f3db013ec60 unknown :-1 s=CLOSED pgs=3068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.423+0000 7f3db6a21640 1 -- 172.21.15.7:0/361000789 >> 172.21.15.7:0/361000789 conn(0x7f3db00b7780 msgr2=0x7f3db00b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:22.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.423+0000 7f3db6a21640 1 -- 172.21.15.7:0/361000789 shutdown_connections 2023-11-25T05:26:22.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.423+0000 7f3db6a21640 1 -- 172.21.15.7:0/361000789 wait complete. 2023-11-25T05:26:22.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.424+0000 7f3db6a21640 10 monclient: build_initial_monmap 2023-11-25T05:26:22.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.424+0000 7f3db6a21640 10 monclient: monmap: 2023-11-25T05:26:22.430 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:22.430 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:22.430 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:22.430 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:22.430 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:22.431 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:22.431 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:22.431 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:22.431 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:22.431 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:22.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.424+0000 7f3db6a21640 1 librados: starting msgr at 2023-11-25T05:26:22.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.424+0000 7f3db6a21640 1 librados: starting objecter 2023-11-25T05:26:22.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.424+0000 7f3db6a21640 1 Processor -- start 2023-11-25T05:26:22.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 1 -- start start 2023-11-25T05:26:22.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 1 librados: setting wanted keys 2023-11-25T05:26:22.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 1 librados: calling monclient init 2023-11-25T05:26:22.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 10 monclient: init 2023-11-25T05:26:22.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:22.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-25T05:26:22.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3db00a8910 0x7f3db0136740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:22.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 10 monclient(hunting): picked mon.noname-b con 0x7f3db00a8910 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:22.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3db0141f90 0x7f3db00ab510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:22.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 10 monclient(hunting): picked mon.noname-c con 0x7f3db0141f90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:22.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3db013a680 0x7f3db013ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:22.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 10 monclient(hunting): picked mon.noname-a con 0x7f3db013a680 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:22.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:22.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db013b980 con 0x7f3db013a680 2023-11-25T05:26:22.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:22.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db013bbf0 con 0x7f3db00a8910 2023-11-25T05:26:22.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:22.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db013bed0 con 0x7f3db0141f90 2023-11-25T05:26:22.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:22.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.425+0000 7f3db6a21640 10 monclient(hunting): authenticate will time out at 2708.668584s 2023-11-25T05:26:22.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.426+0000 7f3daf7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3db0141f90 0x7f3db00ab510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:22.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.426+0000 7f3db4fbe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3db013a680 0x7f3db013ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:22.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.426+0000 7f3daf7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3db0141f90 0x7f3db00ab510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38320/0 (socket says 172.21.15.7:38320) 2023-11-25T05:26:22.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.426+0000 7f3daf7fe640 1 -- 172.21.15.7:0/1570822092 learned_addr learned my addr 172.21.15.7:0/1570822092 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:22.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.426+0000 7f3daf7fe640 10 monclient(hunting): get_auth_request con 0x7f3db0141f90 auth_method 0 2023-11-25T05:26:22.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.426+0000 7f3daf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:22.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.426+0000 7f3daf7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:22.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.426+0000 7f3daf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:22.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.426+0000 7f3db4fbe640 10 monclient(hunting): get_auth_request con 0x7f3db013a680 auth_method 0 2023-11-25T05:26:22.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.426+0000 7f3db4fbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:22.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.426+0000 7f3db4fbe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:22.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.426+0000 7f3db4fbe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:22.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.427+0000 7f3daf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:22.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.427+0000 7f3daf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:22.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.427+0000 7f3daf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:22.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.427+0000 7f3db4fbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:22.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.427+0000 7f3db4fbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:22.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.427+0000 7f3db4fbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:22.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.428+0000 7f3daffff640 1 --2- 172.21.15.7:0/1570822092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3db00a8910 0x7f3db0136740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:22.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.429+0000 7f3daffff640 10 monclient(hunting): get_auth_request con 0x7f3db00a8910 auth_method 0 2023-11-25T05:26:22.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.429+0000 7f3daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:22.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.429+0000 7f3daffff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:22.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.429+0000 7f3daffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:22.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.430+0000 7f3daf7fe640 10 monclient(hunting): handle_auth_done global_id 17405 payload 824 2023-11-25T05:26:22.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.430+0000 7f3daf7fe640 1 -- 172.21.15.7:0/1570822092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3db00a8910 msgr2=0x7f3db0136740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:22.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.430+0000 7f3daf7fe640 1 --2- 172.21.15.7:0/1570822092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3db00a8910 0x7f3db0136740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.430+0000 7f3daf7fe640 1 -- 172.21.15.7:0/1570822092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3db013a680 msgr2=0x7f3db013ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:22.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.430+0000 7f3daf7fe640 1 --2- 172.21.15.7:0/1570822092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3db013a680 0x7f3db013ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.430+0000 7f3daf7fe640 10 monclient: _finish_hunting 0 2023-11-25T05:26:22.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.430+0000 7f3daf7fe640 1 monclient: found mon.noname-c 2023-11-25T05:26:22.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.430+0000 7f3daf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:22.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.430+0000 7f3daf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:26:22.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.430+0000 7f3daf7fe640 1 -- 172.21.15.7:0/1570822092 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da0009400 con 0x7f3db0141f90 2023-11-25T05:26:22.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.432+0000 7f3daf7fe640 1 --2- 172.21.15.7:0/1570822092 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3db0141f90 0x7f3db00ab510 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7f3da400c400 tx=0x7f3da400c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:22.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.432+0000 7f3dacff9640 10 client.?.objecter ms_handle_connect 0x7f3db0141f90 2023-11-25T05:26:22.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.432+0000 7f3dacff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:22.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3dacff9640 1 -- 172.21.15.7:0/1570822092 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da4005ce0 con 0x7f3db0141f90 2023-11-25T05:26:22.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3dacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:22.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3dacff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:26:22.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3dacff9640 10 monclient: dump: 2023-11-25T05:26:22.438 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:22.438 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:22.438 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.438 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.438 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:22.438 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:22.438 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:22.438 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:22.439 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:22.439 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:22.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3dacff9640 10 monclient: _finish_auth 0 2023-11-25T05:26:22.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3dacff9640 10 monclient: _check_auth_tickets 2023-11-25T05:26:22.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3dacff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:22.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3dacff9640 1 -- 172.21.15.7:0/1570822092 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da4005e40 con 0x7f3db0141f90 2023-11-25T05:26:22.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3db6a21640 5 monclient: authenticate success, global_id 17405 2023-11-25T05:26:22.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3db6a21640 10 monclient: _renew_subs 2023-11-25T05:26:22.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3db6a21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:26:22.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3db6a21640 1 -- 172.21.15.7:0/1570822092 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db0137220 con 0x7f3db0141f90 2023-11-25T05:26:22.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3dacff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:22.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3dacff9640 1 -- 172.21.15.7:0/1570822092 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da40072d0 con 0x7f3db0141f90 2023-11-25T05:26:22.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3dacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:22.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3dacff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:26:22.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3dacff9640 10 monclient: dump: 2023-11-25T05:26:22.440 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:22.440 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:22.440 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.440 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:22.440 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:22.440 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:22.440 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:22.440 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:22.440 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:22.440 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:22.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3db6a21640 10 client.17405.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:22.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3db6a21640 10 monclient: _renew_subs 2023-11-25T05:26:22.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3dae7fc640 4 set_mon_vals no callback set 2023-11-25T05:26:22.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3db6a21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:26:22.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3db6a21640 1 -- 172.21.15.7:0/1570822092 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db0137df0 con 0x7f3db0141f90 2023-11-25T05:26:22.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.436+0000 7f3db6a21640 1 librados: init done 2023-11-25T05:26:22.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.437+0000 7f3db6a21640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:22.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.437+0000 7f3db6a21640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:22.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.437+0000 7f3db6a21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:26:22.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.437+0000 7f3db6a21640 1 -- 172.21.15.7:0/1570822092 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db013a3a0 con 0x7f3db0141f90 2023-11-25T05:26:22.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.456+0000 7f3dacff9640 1 -- 172.21.15.7:0/1570822092 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3da40074f0 con 0x7f3db0141f90 2023-11-25T05:26:22.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.457+0000 7f3dacff9640 1 --2- 172.21.15.7:0/1570822092 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f3d9c06c7d0 0x7f3d9c06ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:22.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.457+0000 7f3daffff640 1 --2- 172.21.15.7:0/1570822092 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f3d9c06c7d0 0x7f3d9c06ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:22.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.457+0000 7f3daffff640 10 monclient: get_auth_request con 0x7f3d9c06c7d0 auth_method 0 2023-11-25T05:26:22.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.458+0000 7f3daffff640 1 --2- 172.21.15.7:0/1570822092 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f3d9c06c7d0 0x7f3d9c06ec90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3da0036000 tx=0x7f3da003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:22.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.458+0000 7f3dacff9640 10 client.17405.objecter ms_handle_connect 0x7f3d9c06c7d0 2023-11-25T05:26:22.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.460+0000 7f3dacff9640 1 -- 172.21.15.7:0/1570822092 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f3da4057c70 con 0x7f3db0141f90 2023-11-25T05:26:22.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.460+0000 7f3dacff9640 10 client.17405.objecter ms_dispatch 0x7f3db0000b90 osd_map(918..918 src has 501..918) v4 2023-11-25T05:26:22.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.460+0000 7f3dacff9640 3 client.17405.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:22.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.460+0000 7f3dacff9640 3 client.17405.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:22.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.460+0000 7f3dacff9640 20 client.17405.objecter dump_active .. 0 homeless 2023-11-25T05:26:22.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.715+0000 7f3dacff9640 1 -- 172.21.15.7:0/1570822092 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3da4053fb0 con 0x7f3db0141f90 2023-11-25T05:26:22.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.715+0000 7f3dacff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:22.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.715+0000 7f3dacff9640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:26:22.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.901+0000 7f3db6a21640 1 -- 172.21.15.7:0/1570822092 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f3db014d670 con 0x7f3d9c06c7d0 2023-11-25T05:26:22.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.901+0000 7f3dacff9640 1 -- 172.21.15.7:0/1570822092 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3da003a420 con 0x7f3d9c06c7d0 2023-11-25T05:26:22.903 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 2 to deep-scrub 2023-11-25T05:26:22.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 10 librados: watch_flush enter 2023-11-25T05:26:22.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 10 librados: watch_flush exit 2023-11-25T05:26:22.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 20 client.17405.objecter shutdown clearing up homeless session... 2023-11-25T05:26:22.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 10 client.17405.objecter successfully canceled tick 2023-11-25T05:26:22.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 1 -- 172.21.15.7:0/1570822092 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f3d9c06c7d0 msgr2=0x7f3d9c06ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:22.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 1 --2- 172.21.15.7:0/1570822092 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f3d9c06c7d0 0x7f3d9c06ec90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3da0036000 tx=0x7f3da003c040 comp rx=0 tx=0).stop 2023-11-25T05:26:22.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 10 monclient: shutdown 2023-11-25T05:26:22.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:22.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 1 -- 172.21.15.7:0/1570822092 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3db0141f90 msgr2=0x7f3db00ab510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:22.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 1 --2- 172.21.15.7:0/1570822092 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3db0141f90 0x7f3db00ab510 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7f3da400c400 tx=0x7f3da400c8c0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 1 -- 172.21.15.7:0/1570822092 shutdown_connections 2023-11-25T05:26:22.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 1 --2- 172.21.15.7:0/1570822092 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f3d9c06c7d0 0x7f3d9c06ec90 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 1 --2- 172.21.15.7:0/1570822092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3db013a680 0x7f3db013ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 1 --2- 172.21.15.7:0/1570822092 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3db0141f90 0x7f3db00ab510 unknown :-1 s=CLOSED pgs=3069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 1 --2- 172.21.15.7:0/1570822092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3db00a8910 0x7f3db0136740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:22.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 1 -- 172.21.15.7:0/1570822092 >> 172.21.15.7:0/1570822092 conn(0x7f3db00b7780 msgr2=0x7f3db013c860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:22.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 1 -- 172.21.15.7:0/1570822092 shutdown_connections 2023-11-25T05:26:22.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 1 -- 172.21.15.7:0/1570822092 wait complete. 2023-11-25T05:26:22.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:22.904+0000 7f3db6a21640 1 librados: shutdown 2023-11-25T05:26:22.919 INFO:tasks.ceph:Scrubbing osd.3 2023-11-25T05:26:22.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-11-25T05:26:23.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.034+0000 7fed27183640 10 monclient: get_monmap_and_config 2023-11-25T05:26:23.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.034+0000 7fed27183640 10 monclient: build_initial_monmap 2023-11-25T05:26:23.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.034+0000 7fed27183640 10 monclient: monmap: 2023-11-25T05:26:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:23.035257+0000 2023-11-25T05:26:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:23.035257+0000 2023-11-25T05:26:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:23.036 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.035+0000 7fed27183640 1 Processor -- start 2023-11-25T05:26:23.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.035+0000 7fed27183640 1 -- start start 2023-11-25T05:26:23.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.035+0000 7fed27183640 10 monclient: init 2023-11-25T05:26:23.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.035+0000 7fed27183640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:23.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.035+0000 7fed27183640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-11-25T05:26:23.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.035+0000 7fed27183640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fed200711b0 0x7fed20071590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.035+0000 7fed27183640 10 monclient(hunting): picked mon.noname-b con 0x7fed200711b0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.035+0000 7fed27183640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed20070400 addr v1:172.21.15.7:6789/0 2023-11-25T05:26:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.035+0000 7fed27183640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed2006c240 addr v1:172.21.15.90:6789/0 2023-11-25T05:26:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.035+0000 7fed27183640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed200afca0 con 0x7fed20070400 2023-11-25T05:26:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.035+0000 7fed27183640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2006be20 con 0x7fed2006c240 2023-11-25T05:26:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.035+0000 7fed27183640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.035+0000 7fed27183640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed2013f9c0 con 0x7fed200711b0 2023-11-25T05:26:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.035+0000 7fed27183640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.035+0000 7fed27183640 10 monclient(hunting): authenticate will time out at 2709.278621s 2023-11-25T05:26:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.036+0000 7fed24f1f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fed200711b0 0x7fed20071590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.036+0000 7fed1ffff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fed20070400 0x7fed2013f2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49918/0 (socket says 172.21.15.7:49918) 2023-11-25T05:26:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.036+0000 7fed1ffff640 1 -- 172.21.15.7:0/3757293228 learned_addr learned my addr 172.21.15.7:0/3757293228 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:26:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.036+0000 7fed24f1f640 10 monclient(hunting): get_auth_request con 0x7fed200711b0 auth_method 0 2023-11-25T05:26:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.036+0000 7fed24f1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.036+0000 7fed24f1f640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.036+0000 7fed24f1f640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.037+0000 7fed24f1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.037+0000 7fed24f1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.037+0000 7fed24f1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:23.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.038+0000 7fed1f7fe640 1 -- 172.21.15.7:0/3757293228 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888273151 0 0) 0x7fed2006be20 con 0x7fed2006c240 2023-11-25T05:26:23.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.038+0000 7fed1f7fe640 10 monclient(hunting): _negotiate 2023-11-25T05:26:23.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.038+0000 7fed1f7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.038+0000 7fed1f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.038+0000 7fed1f7fe640 10 monclient(hunting): my global_id is 21256 2023-11-25T05:26:23.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.038+0000 7fed1f7fe640 1 -- 172.21.15.7:0/3757293228 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed0c003520 con 0x7fed2006c240 2023-11-25T05:26:23.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.039+0000 7fed24f1f640 10 monclient(hunting): handle_auth_done global_id 17408 payload 274 2023-11-25T05:26:23.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.039+0000 7fed24f1f640 1 -- 172.21.15.7:0/3757293228 >> v1:172.21.15.90:6789/0 conn(0x7fed2006c240 legacy=0x7fed2006c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.039+0000 7fed24f1f640 1 -- 172.21.15.7:0/3757293228 >> v1:172.21.15.7:6789/0 conn(0x7fed20070400 legacy=0x7fed2013f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.039+0000 7fed24f1f640 10 monclient: _finish_hunting 0 2023-11-25T05:26:23.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.039+0000 7fed24f1f640 1 monclient: found mon.noname-b 2023-11-25T05:26:23.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.039+0000 7fed24f1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:23.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.039+0000 7fed24f1f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:26:23.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.039+0000 7fed24f1f640 1 -- 172.21.15.7:0/3757293228 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2006cdc0 con 0x7fed200711b0 2023-11-25T05:26:23.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.041+0000 7fed24f1f640 1 --2- 172.21.15.7:0/3757293228 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fed200711b0 0x7fed20071590 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fed100097d0 tx=0x7fed1002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9297ced18eb95ef8 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:23.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.044+0000 7fed1f7fe640 1 -- 172.21.15.7:0/3757293228 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed10031890 con 0x7fed200711b0 2023-11-25T05:26:23.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.044+0000 7fed1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:23.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed1f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:23.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed1f7fe640 10 monclient: dump: 2023-11-25T05:26:23.046 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:23.047 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:23.047 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.047 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.047 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:23.047 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.047 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:23.047 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:23.047 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:23.047 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed1f7fe640 10 monclient: _finish_auth 0 2023-11-25T05:26:23.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed1f7fe640 10 monclient: _check_auth_tickets 2023-11-25T05:26:23.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed1f7fe640 1 -- 172.21.15.7:0/3757293228 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed100319f0 con 0x7fed200711b0 2023-11-25T05:26:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed1f7fe640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed1f7fe640 1 -- 172.21.15.7:0/3757293228 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed10031d80 con 0x7fed200711b0 2023-11-25T05:26:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed27183640 5 monclient: authenticate success, global_id 17408 2023-11-25T05:26:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed27183640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed27183640 4 set_mon_vals no callback set 2023-11-25T05:26:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed1f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:26:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed1f7fe640 10 monclient: dump: 2023-11-25T05:26:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:23.049 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.049 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.049 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:23.049 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.049 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:23.049 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:23.049 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:23.049 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed27183640 10 monclient: shutdown 2023-11-25T05:26:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed27183640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed27183640 1 -- 172.21.15.7:0/3757293228 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fed200711b0 msgr2=0x7fed20071590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed27183640 1 --2- 172.21.15.7:0/3757293228 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fed200711b0 0x7fed20071590 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fed100097d0 tx=0x7fed1002b930 comp rx=0 tx=0).stop 2023-11-25T05:26:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed27183640 1 -- 172.21.15.7:0/3757293228 shutdown_connections 2023-11-25T05:26:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed27183640 1 --2- 172.21.15.7:0/3757293228 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fed200711b0 0x7fed20071590 unknown :-1 s=CLOSED pgs=3070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed27183640 1 -- 172.21.15.7:0/3757293228 >> 172.21.15.7:0/3757293228 conn(0x7fed200b5880 msgr2=0x7fed200b5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed27183640 1 -- 172.21.15.7:0/3757293228 shutdown_connections 2023-11-25T05:26:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.045+0000 7fed27183640 1 -- 172.21.15.7:0/3757293228 wait complete. 2023-11-25T05:26:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.046+0000 7fed27183640 10 monclient: build_initial_monmap 2023-11-25T05:26:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.046+0000 7fed27183640 10 monclient: monmap: 2023-11-25T05:26:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:23.054 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.046+0000 7fed27183640 1 librados: starting msgr at 2023-11-25T05:26:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.046+0000 7fed27183640 1 librados: starting objecter 2023-11-25T05:26:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.046+0000 7fed27183640 1 Processor -- start 2023-11-25T05:26:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.046+0000 7fed27183640 1 -- start start 2023-11-25T05:26:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.046+0000 7fed27183640 1 librados: setting wanted keys 2023-11-25T05:26:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.046+0000 7fed27183640 1 librados: calling monclient init 2023-11-25T05:26:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.046+0000 7fed27183640 10 monclient: init 2023-11-25T05:26:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:26:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fed2006c240 0x7fed201410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 10 monclient(hunting): picked mon.noname-c con 0x7fed2006c240 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed20070400 0x7fed200b4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 10 monclient(hunting): picked mon.noname-b con 0x7fed20070400 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed200711b0 0x7fed20141b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 10 monclient(hunting): picked mon.noname-a con 0x7fed200711b0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed201420b0 con 0x7fed200711b0 2023-11-25T05:26:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed20142290 con 0x7fed20070400 2023-11-25T05:26:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed201309e0 con 0x7fed2006c240 2023-11-25T05:26:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.047+0000 7fed27183640 10 monclient(hunting): authenticate will time out at 2709.290471s 2023-11-25T05:26:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed24f1f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fed2006c240 0x7fed201410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed1ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed20070400 0x7fed200b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed25720640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed200711b0 0x7fed20141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed24f1f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fed2006c240 0x7fed201410d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38338/0 (socket says 172.21.15.7:38338) 2023-11-25T05:26:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed24f1f640 1 -- 172.21.15.7:0/1696676599 learned_addr learned my addr 172.21.15.7:0/1696676599 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed24f1f640 10 monclient(hunting): get_auth_request con 0x7fed2006c240 auth_method 0 2023-11-25T05:26:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed24f1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed24f1f640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed24f1f640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed1ffff640 10 monclient(hunting): get_auth_request con 0x7fed20070400 auth_method 0 2023-11-25T05:26:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed1ffff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed1ffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed25720640 10 monclient(hunting): get_auth_request con 0x7fed200711b0 auth_method 0 2023-11-25T05:26:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed25720640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed25720640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.048+0000 7fed25720640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.049+0000 7fed1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.049+0000 7fed1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.049+0000 7fed1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.049+0000 7fed24f1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.049+0000 7fed24f1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.049+0000 7fed24f1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.049+0000 7fed25720640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.049+0000 7fed25720640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.049+0000 7fed25720640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.051+0000 7fed1ffff640 10 monclient(hunting): handle_auth_done global_id 21259 payload 824 2023-11-25T05:26:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.051+0000 7fed1ffff640 1 -- 172.21.15.7:0/1696676599 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fed2006c240 msgr2=0x7fed201410d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.051+0000 7fed1ffff640 1 --2- 172.21.15.7:0/1696676599 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fed2006c240 0x7fed201410d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.051+0000 7fed1ffff640 1 -- 172.21.15.7:0/1696676599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed200711b0 msgr2=0x7fed20141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.051+0000 7fed1ffff640 1 --2- 172.21.15.7:0/1696676599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed200711b0 0x7fed20141b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.051+0000 7fed1ffff640 10 monclient: _finish_hunting 0 2023-11-25T05:26:23.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.051+0000 7fed1ffff640 1 monclient: found mon.noname-b 2023-11-25T05:26:23.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.051+0000 7fed1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:23.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.051+0000 7fed1ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:26:23.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.051+0000 7fed1ffff640 1 -- 172.21.15.7:0/1696676599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed10009400 con 0x7fed20070400 2023-11-25T05:26:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.053+0000 7fed1ffff640 1 --2- 172.21.15.7:0/1696676599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed20070400 0x7fed200b4730 secure :-1 s=READY pgs=5157 cs=0 l=1 rev1=1 crypto rx=0x7fed0800c370 tx=0x7fed0800c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.053+0000 7fed1d7fa640 10 client.?.objecter ms_handle_connect 0x7fed20070400 2023-11-25T05:26:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.053+0000 7fed1d7fa640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed1d7fa640 1 -- 172.21.15.7:0/1696676599 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08005ce0 con 0x7fed20070400 2023-11-25T05:26:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed1d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed1d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed1d7fa640 10 monclient: dump: 2023-11-25T05:26:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:23.062 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed1d7fa640 10 monclient: _finish_auth 0 2023-11-25T05:26:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed1d7fa640 10 monclient: _check_auth_tickets 2023-11-25T05:26:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed1d7fa640 1 -- 172.21.15.7:0/1696676599 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed08005e40 con 0x7fed20070400 2023-11-25T05:26:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed1d7fa640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed1d7fa640 1 -- 172.21.15.7:0/1696676599 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08007310 con 0x7fed20070400 2023-11-25T05:26:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed27183640 5 monclient: authenticate success, global_id 21259 2023-11-25T05:26:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed1d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed1d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed1d7fa640 10 monclient: dump: 2023-11-25T05:26:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:23.064 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed27183640 10 monclient: _renew_subs 2023-11-25T05:26:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed1effd640 4 set_mon_vals no callback set 2023-11-25T05:26:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed27183640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed27183640 1 -- 172.21.15.7:0/1696676599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed20130c60 con 0x7fed20070400 2023-11-25T05:26:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed27183640 10 client.21259.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed27183640 10 monclient: _renew_subs 2023-11-25T05:26:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed27183640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed27183640 1 -- 172.21.15.7:0/1696676599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed20131120 con 0x7fed20070400 2023-11-25T05:26:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.057+0000 7fed27183640 1 librados: init done 2023-11-25T05:26:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.058+0000 7fed27183640 10 client.21259.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.058+0000 7fed27183640 15 client.21259.objecter _session_command_op_assign -1 1 2023-11-25T05:26:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.058+0000 7fed27183640 20 client.21259.objecter _get_session osd=-1 returning homeless 2023-11-25T05:26:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.058+0000 7fed27183640 10 client.21259.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.058+0000 7fed27183640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.058+0000 7fed27183640 1 -- 172.21.15.7:0/1696676599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fed20131390 con 0x7fed20070400 2023-11-25T05:26:23.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.077+0000 7fed1d7fa640 1 -- 172.21.15.7:0/1696676599 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fed08007470 con 0x7fed20070400 2023-11-25T05:26:23.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.077+0000 7fed1d7fa640 1 --2- 172.21.15.7:0/1696676599 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fed0006c7d0 0x7fed0006ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.077+0000 7fed24f1f640 1 --2- 172.21.15.7:0/1696676599 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fed0006c7d0 0x7fed0006ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.077+0000 7fed24f1f640 10 monclient: get_auth_request con 0x7fed0006c7d0 auth_method 0 2023-11-25T05:26:23.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.078+0000 7fed1d7fa640 10 client.21259.objecter ms_handle_connect 0x7fed0006c7d0 2023-11-25T05:26:23.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.078+0000 7fed24f1f640 1 --2- 172.21.15.7:0/1696676599 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fed0006c7d0 0x7fed0006ec90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fed10000c00 tx=0x7fed1003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:23.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 1 -- 172.21.15.7:0/1696676599 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7fed08007720 con 0x7fed20070400 2023-11-25T05:26:23.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 10 client.21259.objecter ms_dispatch 0x7fed20000b90 osd_map(918..918 src has 501..918) v4 2023-11-25T05:26:23.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 3 client.21259.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:23.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 3 client.21259.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:23.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 10 client.21259.objecter checking command 1 2023-11-25T05:26:23.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 1 --2- 172.21.15.7:0/1696676599 >> [v2:172.21.15.7:6816/59754388,v1:172.21.15.7:6817/59754388] conn(0x7fed00074050 0x7fed000764f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 20 client.21259.objecter _get_session s=0x7fed00073980 osd=3 3 2023-11-25T05:26:23.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 20 client.21259.objecter put_session s=0x7fed00073980 osd=3 3 2023-11-25T05:26:23.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 15 client.21259.objecter _session_command_op_remove -1 1 2023-11-25T05:26:23.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 20 client.21259.objecter _get_session s=0x7fed00073980 osd=3 3 2023-11-25T05:26:23.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 20 client.21259.objecter get_session s=0x7fed00073980 osd=3 3 2023-11-25T05:26:23.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 15 client.21259.objecter _session_command_op_assign 3 1 2023-11-25T05:26:23.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 20 client.21259.objecter put_session s=0x7fed00073980 osd=3 4 2023-11-25T05:26:23.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 10 client.21259.objecter _send_command 1 2023-11-25T05:26:23.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 1 -- 172.21.15.7:0/1696676599 --> [v2:172.21.15.7:6816/59754388,v1:172.21.15.7:6817/59754388] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed00076bc0 con 0x7fed00074050 2023-11-25T05:26:23.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.079+0000 7fed1d7fa640 20 client.21259.objecter dump_active .. 0 homeless 2023-11-25T05:26:23.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.080+0000 7fed25720640 1 --2- 172.21.15.7:0/1696676599 >> [v2:172.21.15.7:6816/59754388,v1:172.21.15.7:6817/59754388] conn(0x7fed00074050 0x7fed000764f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.080+0000 7fed25720640 10 monclient: get_auth_request con 0x7fed00074050 auth_method 0 2023-11-25T05:26:23.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.081+0000 7fed1d7fa640 1 -- 172.21.15.7:0/1696676599 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=918) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed08054330 con 0x7fed20070400 2023-11-25T05:26:23.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.081+0000 7fed1d7fa640 10 monclient: handle_get_version_reply finishing 1 version 918 2023-11-25T05:26:23.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.082+0000 7fed25720640 1 --2- 172.21.15.7:0/1696676599 >> [v2:172.21.15.7:6816/59754388,v1:172.21.15.7:6817/59754388] conn(0x7fed00074050 0x7fed000764f0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:23.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.082+0000 7fed1d7fa640 10 client.21259.objecter ms_handle_connect 0x7fed00074050 2023-11-25T05:26:23.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.108+0000 7fed1d7fa640 1 -- 172.21.15.7:0/1696676599 <== osd.3 v2:172.21.15.7:6816/59754388 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fed00076bc0 con 0x7fed00074050 2023-11-25T05:26:23.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.108+0000 7fed1d7fa640 10 client.21259.objecter ms_dispatch 0x7fed20000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:26:23.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.108+0000 7fed1d7fa640 10 client.21259.objecter _finish_command 1 = system:0 2023-11-25T05:26:23.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.108+0000 7fed1d7fa640 20 client.21259.objecter put_session s=0x7fed00073980 osd=3 4 2023-11-25T05:26:23.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.108+0000 7fed1d7fa640 15 client.21259.objecter _session_command_op_remove 3 1 2023-11-25T05:26:23.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.121+0000 7fed27183640 10 client.21259.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-11-25T05:26:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.121+0000 7fed27183640 15 client.21259.objecter _session_command_op_assign -1 2 2023-11-25T05:26:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.121+0000 7fed27183640 20 client.21259.objecter _get_session s=0x7fed00073980 osd=3 3 2023-11-25T05:26:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.121+0000 7fed27183640 20 client.21259.objecter put_session s=0x7fed00073980 osd=3 3 2023-11-25T05:26:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.121+0000 7fed27183640 20 client.21259.objecter _get_session s=0x7fed00073980 osd=3 3 2023-11-25T05:26:23.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.121+0000 7fed27183640 15 client.21259.objecter _session_command_op_remove -1 2 2023-11-25T05:26:23.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.121+0000 7fed27183640 20 client.21259.objecter get_session s=0x7fed00073980 osd=3 3 2023-11-25T05:26:23.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.121+0000 7fed27183640 15 client.21259.objecter _session_command_op_assign 3 2 2023-11-25T05:26:23.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.121+0000 7fed27183640 20 client.21259.objecter put_session s=0x7fed00073980 osd=3 4 2023-11-25T05:26:23.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.121+0000 7fed27183640 10 client.21259.objecter _send_command 2 2023-11-25T05:26:23.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.121+0000 7fed27183640 1 -- 172.21.15.7:0/1696676599 --> [v2:172.21.15.7:6816/59754388,v1:172.21.15.7:6817/59754388] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fed20075af0 con 0x7fed00074050 2023-11-25T05:26:23.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed1d7fa640 1 -- 172.21.15.7:0/1696676599 <== osd.3 v2:172.21.15.7:6816/59754388 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fed20075af0 con 0x7fed00074050 2023-11-25T05:26:23.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed1d7fa640 10 client.21259.objecter ms_dispatch 0x7fed20000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:26:23.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed1d7fa640 10 client.21259.objecter _finish_command 2 = system:0 2023-11-25T05:26:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed1d7fa640 20 client.21259.objecter put_session s=0x7fed00073980 osd=3 4 2023-11-25T05:26:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed1d7fa640 15 client.21259.objecter _session_command_op_remove 3 2 2023-11-25T05:26:23.139 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-11-25T05:26:23.139 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-25T05:26:23.139 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-11-25T05:26:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 10 librados: watch_flush enter 2023-11-25T05:26:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 10 librados: watch_flush exit 2023-11-25T05:26:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 10 client.21259.objecter close_session for osd.3 2023-11-25T05:26:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 1 -- 172.21.15.7:0/1696676599 >> [v2:172.21.15.7:6816/59754388,v1:172.21.15.7:6817/59754388] conn(0x7fed00074050 msgr2=0x7fed000764f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 1 --2- 172.21.15.7:0/1696676599 >> [v2:172.21.15.7:6816/59754388,v1:172.21.15.7:6817/59754388] conn(0x7fed00074050 0x7fed000764f0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 20 client.21259.objecter put_session s=0x7fed00073980 osd=3 1 2023-11-25T05:26:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 20 client.21259.objecter shutdown clearing up homeless session... 2023-11-25T05:26:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 10 client.21259.objecter successfully canceled tick 2023-11-25T05:26:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 1 -- 172.21.15.7:0/1696676599 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fed0006c7d0 msgr2=0x7fed0006ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 1 --2- 172.21.15.7:0/1696676599 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fed0006c7d0 0x7fed0006ec90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fed10000c00 tx=0x7fed1003c040 comp rx=0 tx=0).stop 2023-11-25T05:26:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 10 monclient: shutdown 2023-11-25T05:26:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:23.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 1 -- 172.21.15.7:0/1696676599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed20070400 msgr2=0x7fed200b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 1 --2- 172.21.15.7:0/1696676599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed20070400 0x7fed200b4730 secure :-1 s=READY pgs=5157 cs=0 l=1 rev1=1 crypto rx=0x7fed0800c370 tx=0x7fed0800c830 comp rx=0 tx=0).stop 2023-11-25T05:26:23.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed25720640 1 -- 172.21.15.7:0/1696676599 reap_dead start 2023-11-25T05:26:23.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 1 -- 172.21.15.7:0/1696676599 shutdown_connections 2023-11-25T05:26:23.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.137+0000 7fed27183640 1 -- 172.21.15.7:0/1696676599 >> 172.21.15.7:0/1696676599 conn(0x7fed200b5880 msgr2=0x7fed20133fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:23.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.138+0000 7fed27183640 1 -- 172.21.15.7:0/1696676599 shutdown_connections 2023-11-25T05:26:23.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.138+0000 7fed27183640 1 -- 172.21.15.7:0/1696676599 wait complete. 2023-11-25T05:26:23.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.138+0000 7fed27183640 1 librados: shutdown 2023-11-25T05:26:23.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-11-25T05:26:23.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.260+0000 7f33ab11d640 10 monclient: get_monmap_and_config 2023-11-25T05:26:23.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.260+0000 7f33ab11d640 10 monclient: build_initial_monmap 2023-11-25T05:26:23.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.260+0000 7f33ab11d640 10 monclient: monmap: 2023-11-25T05:26:23.265 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:23.265 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:23.265 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:23.261713+0000 2023-11-25T05:26:23.265 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:23.261713+0000 2023-11-25T05:26:23.266 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:23.266 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.266 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:23.266 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:23.266 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:23.266 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:23.266 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:23.266 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.261+0000 7f33ab11d640 1 Processor -- start 2023-11-25T05:26:23.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.261+0000 7f33ab11d640 1 -- start start 2023-11-25T05:26:23.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.261+0000 7f33ab11d640 10 monclient: init 2023-11-25T05:26:23.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.262+0000 7f33ab11d640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:23.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.262+0000 7f33ab11d640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2023-11-25T05:26:23.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.262+0000 7f33ab11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f33a406ccd0 addr v1:172.21.15.90:6789/0 2023-11-25T05:26:23.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.262+0000 7f33ab11d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33a406c0e0 0x7f33a4071700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.262+0000 7f33ab11d640 10 monclient(hunting): picked mon.noname-b con 0x7f33a406c0e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:23.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.262+0000 7f33ab11d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f33a406d0b0 addr v1:172.21.15.7:6789/0 2023-11-25T05:26:23.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.262+0000 7f33ab11d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a40afb80 con 0x7f33a406d0b0 2023-11-25T05:26:23.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.262+0000 7f33ab11d640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a4071070 con 0x7f33a406ccd0 2023-11-25T05:26:23.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.262+0000 7f33ab11d640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:23.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.262+0000 7f33ab11d640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a40b0010 con 0x7f33a406c0e0 2023-11-25T05:26:23.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.262+0000 7f33ab11d640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:23.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.262+0000 7f33ab11d640 10 monclient(hunting): authenticate will time out at 2709.505061s 2023-11-25T05:26:23.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.262+0000 7f33a3fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33a406c0e0 0x7f33a4071700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.262+0000 7f33a8eb9640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f33a406ccd0 0x7f33a413f110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:40578/0 (socket says 172.21.15.7:40578) 2023-11-25T05:26:23.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.262+0000 7f33a8eb9640 1 -- 172.21.15.7:0/2534881805 learned_addr learned my addr 172.21.15.7:0/2534881805 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:26:23.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.263+0000 7f33a3fff640 10 monclient(hunting): get_auth_request con 0x7f33a406c0e0 auth_method 0 2023-11-25T05:26:23.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.263+0000 7f33a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:23.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.263+0000 7f33a3fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.263+0000 7f33a3fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.263+0000 7f33a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:23.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.263+0000 7f33a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:23.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.263+0000 7f33a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:23.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a37fe640 1 -- 172.21.15.7:0/2534881805 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084550501 0 0) 0x7f33a4071070 con 0x7f33a406ccd0 2023-11-25T05:26:23.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a37fe640 10 monclient(hunting): _negotiate 2023-11-25T05:26:23.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a37fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a37fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a37fe640 10 monclient(hunting): my global_id is 21262 2023-11-25T05:26:23.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a37fe640 1 -- 172.21.15.7:0/2534881805 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f338c003520 con 0x7f33a406ccd0 2023-11-25T05:26:23.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a37fe640 1 -- 172.21.15.7:0/2534881805 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596216366 0 0) 0x7f33a40afb80 con 0x7f33a406d0b0 2023-11-25T05:26:23.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a37fe640 10 monclient(hunting): _negotiate 2023-11-25T05:26:23.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a37fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a37fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a37fe640 10 monclient(hunting): my global_id is 15936 2023-11-25T05:26:23.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a37fe640 1 -- 172.21.15.7:0/2534881805 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a4071070 con 0x7f33a406d0b0 2023-11-25T05:26:23.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a3fff640 10 monclient(hunting): handle_auth_done global_id 17414 payload 274 2023-11-25T05:26:23.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a3fff640 1 -- 172.21.15.7:0/2534881805 >> v1:172.21.15.90:6789/0 conn(0x7f33a406ccd0 legacy=0x7f33a413f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a3fff640 1 -- 172.21.15.7:0/2534881805 >> v1:172.21.15.7:6789/0 conn(0x7f33a406d0b0 legacy=0x7f33a4071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a3fff640 10 monclient: _finish_hunting 0 2023-11-25T05:26:23.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a3fff640 1 monclient: found mon.noname-b 2023-11-25T05:26:23.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:23.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:26:23.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.265+0000 7f33a3fff640 1 -- 172.21.15.7:0/2534881805 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a40712e0 con 0x7f33a406c0e0 2023-11-25T05:26:23.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.268+0000 7f33a3fff640 1 --2- 172.21.15.7:0/2534881805 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33a406c0e0 0x7f33a4071700 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7f33980097d0 tx=0x7f339802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1b0346c07c63544 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:23.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33a37fe640 1 -- 172.21.15.7:0/2534881805 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3398031800 con 0x7f33a406c0e0 2023-11-25T05:26:23.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:23.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:23.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33a37fe640 10 monclient: dump: 2023-11-25T05:26:23.274 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:23.274 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:23.274 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.274 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.274 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:23.274 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.275 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:23.275 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:23.275 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:23.275 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33a37fe640 10 monclient: _finish_auth 0 2023-11-25T05:26:23.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33a37fe640 10 monclient: _check_auth_tickets 2023-11-25T05:26:23.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:23.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33a37fe640 1 -- 172.21.15.7:0/2534881805 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3398031960 con 0x7f33a406c0e0 2023-11-25T05:26:23.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33ab11d640 5 monclient: authenticate success, global_id 17414 2023-11-25T05:26:23.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33ab11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:26:23.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33a37fe640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:23.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33ab11d640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:23.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33ab11d640 4 set_mon_vals no callback set 2023-11-25T05:26:23.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33a37fe640 1 -- 172.21.15.7:0/2534881805 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3398031cf0 con 0x7f33a406c0e0 2023-11-25T05:26:23.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:23.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:26:23.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.272+0000 7f33a37fe640 10 monclient: dump: 2023-11-25T05:26:23.276 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:23.276 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:23.276 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.276 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.276 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:23.276 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.276 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:23.276 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:23.276 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:23.276 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.273+0000 7f33ab11d640 10 monclient: shutdown 2023-11-25T05:26:23.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.273+0000 7f33ab11d640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:23.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.273+0000 7f33ab11d640 1 -- 172.21.15.7:0/2534881805 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33a406c0e0 msgr2=0x7f33a4071700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.273+0000 7f33ab11d640 1 --2- 172.21.15.7:0/2534881805 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33a406c0e0 0x7f33a4071700 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7f33980097d0 tx=0x7f339802b8a0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.273+0000 7f33ab11d640 1 -- 172.21.15.7:0/2534881805 shutdown_connections 2023-11-25T05:26:23.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.273+0000 7f33ab11d640 1 --2- 172.21.15.7:0/2534881805 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33a406c0e0 0x7f33a4071700 unknown :-1 s=CLOSED pgs=3071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.273+0000 7f33ab11d640 1 -- 172.21.15.7:0/2534881805 >> 172.21.15.7:0/2534881805 conn(0x7f33a40b5ca0 msgr2=0x7f33a40b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:23.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.273+0000 7f33ab11d640 1 -- 172.21.15.7:0/2534881805 shutdown_connections 2023-11-25T05:26:23.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.273+0000 7f33ab11d640 1 -- 172.21.15.7:0/2534881805 wait complete. 2023-11-25T05:26:23.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.274+0000 7f33ab11d640 10 monclient: build_initial_monmap 2023-11-25T05:26:23.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.274+0000 7f33ab11d640 10 monclient: monmap: 2023-11-25T05:26:23.283 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:23.283 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:23.283 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:23.283 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:23.283 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:23.283 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.283 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:23.286 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:23.286 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:23.286 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.274+0000 7f33ab11d640 1 librados: starting msgr at 2023-11-25T05:26:23.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.274+0000 7f33ab11d640 1 librados: starting objecter 2023-11-25T05:26:23.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.274+0000 7f33ab11d640 1 Processor -- start 2023-11-25T05:26:23.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.274+0000 7f33ab11d640 1 -- start start 2023-11-25T05:26:23.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.274+0000 7f33ab11d640 1 librados: setting wanted keys 2023-11-25T05:26:23.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.274+0000 7f33ab11d640 1 librados: calling monclient init 2023-11-25T05:26:23.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.274+0000 7f33ab11d640 10 monclient: init 2023-11-25T05:26:23.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:23.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:26:23.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33a406c0e0 0x7f33a40aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 10 monclient(hunting): picked mon.noname-c con 0x7f33a406c0e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:23.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33a406ccd0 0x7f33a4136d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 10 monclient(hunting): picked mon.noname-b con 0x7f33a406ccd0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:23.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33a406d0b0 0x7f33a413fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 10 monclient(hunting): picked mon.noname-a con 0x7f33a406d0b0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:23.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:23.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a41372a0 con 0x7f33a406d0b0 2023-11-25T05:26:23.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:23.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a4136860 con 0x7f33a406ccd0 2023-11-25T05:26:23.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:23.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a4138110 con 0x7f33a406c0e0 2023-11-25T05:26:23.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:23.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.275+0000 7f33ab11d640 10 monclient(hunting): authenticate will time out at 2709.518415s 2023-11-25T05:26:23.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a8eb9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33a406c0e0 0x7f33a40aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33a406ccd0 0x7f33a4136d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a96ba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33a406d0b0 0x7f33a413fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a8eb9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33a406c0e0 0x7f33a40aea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38366/0 (socket says 172.21.15.7:38366) 2023-11-25T05:26:23.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a8eb9640 1 -- 172.21.15.7:0/939092817 learned_addr learned my addr 172.21.15.7:0/939092817 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:23.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a8eb9640 10 monclient(hunting): get_auth_request con 0x7f33a406c0e0 auth_method 0 2023-11-25T05:26:23.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a8eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:23.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a8eb9640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a8eb9640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a96ba640 10 monclient(hunting): get_auth_request con 0x7f33a406d0b0 auth_method 0 2023-11-25T05:26:23.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a96ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:23.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a96ba640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a96ba640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a3fff640 10 monclient(hunting): get_auth_request con 0x7f33a406ccd0 auth_method 0 2023-11-25T05:26:23.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:23.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a3fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a3fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a8eb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:23.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a8eb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:23.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.276+0000 7f33a8eb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:23.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.277+0000 7f33a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:23.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.277+0000 7f33a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:23.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.277+0000 7f33a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:23.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.277+0000 7f33a96ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:23.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.277+0000 7f33a96ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:23.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.277+0000 7f33a96ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:23.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.279+0000 7f33a3fff640 10 monclient(hunting): handle_auth_done global_id 21265 payload 824 2023-11-25T05:26:23.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.279+0000 7f33a3fff640 1 -- 172.21.15.7:0/939092817 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33a406c0e0 msgr2=0x7f33a40aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.279+0000 7f33a3fff640 1 --2- 172.21.15.7:0/939092817 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33a406c0e0 0x7f33a40aea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.279+0000 7f33a3fff640 1 -- 172.21.15.7:0/939092817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33a406d0b0 msgr2=0x7f33a413fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.279+0000 7f33a3fff640 1 --2- 172.21.15.7:0/939092817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33a406d0b0 0x7f33a413fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.279+0000 7f33a3fff640 10 monclient: _finish_hunting 0 2023-11-25T05:26:23.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.279+0000 7f33a3fff640 1 monclient: found mon.noname-b 2023-11-25T05:26:23.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.279+0000 7f33a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:23.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.279+0000 7f33a3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:26:23.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.279+0000 7f33a3fff640 1 -- 172.21.15.7:0/939092817 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3398009400 con 0x7f33a406ccd0 2023-11-25T05:26:23.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.281+0000 7f33a3fff640 1 --2- 172.21.15.7:0/939092817 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33a406ccd0 0x7f33a4136d60 secure :-1 s=READY pgs=5159 cs=0 l=1 rev1=1 crypto rx=0x7f3398033000 tx=0x7f33980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:23.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.282+0000 7f33a17fa640 10 client.?.objecter ms_handle_connect 0x7f33a406ccd0 2023-11-25T05:26:23.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.282+0000 7f33a17fa640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:23.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33a17fa640 1 -- 172.21.15.7:0/939092817 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339803bc70 con 0x7f33a406ccd0 2023-11-25T05:26:23.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:23.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33a17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:23.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33a17fa640 10 monclient: dump: 2023-11-25T05:26:23.299 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:23.299 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:23.299 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.299 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.299 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:23.299 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.299 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:23.299 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:23.299 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:23.299 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33a17fa640 10 monclient: _finish_auth 0 2023-11-25T05:26:23.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33a17fa640 10 monclient: _check_auth_tickets 2023-11-25T05:26:23.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:23.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33a17fa640 1 -- 172.21.15.7:0/939092817 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339803be10 con 0x7f33a406ccd0 2023-11-25T05:26:23.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33a17fa640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:23.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33a17fa640 1 -- 172.21.15.7:0/939092817 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3398045630 con 0x7f33a406ccd0 2023-11-25T05:26:23.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:23.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:23.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33a17fa640 10 monclient: dump: 2023-11-25T05:26:23.300 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:23.300 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:23.301 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.301 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.301 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:23.301 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.301 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:23.301 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:23.301 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:23.301 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33ab11d640 5 monclient: authenticate success, global_id 21265 2023-11-25T05:26:23.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33ab11d640 10 monclient: _renew_subs 2023-11-25T05:26:23.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33ab11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:23.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33ab11d640 1 -- 172.21.15.7:0/939092817 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a41383f0 con 0x7f33a406ccd0 2023-11-25T05:26:23.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33a2ffd640 4 set_mon_vals no callback set 2023-11-25T05:26:23.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33ab11d640 10 client.21265.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:23.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33ab11d640 10 monclient: _renew_subs 2023-11-25T05:26:23.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33ab11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:23.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33ab11d640 1 -- 172.21.15.7:0/939092817 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a41401e0 con 0x7f33a406ccd0 2023-11-25T05:26:23.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.285+0000 7f33ab11d640 1 librados: init done 2023-11-25T05:26:23.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.286+0000 7f33ab11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:23.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.286+0000 7f33ab11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:23.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.286+0000 7f33ab11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:23.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.286+0000 7f33ab11d640 1 -- 172.21.15.7:0/939092817 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a4136a80 con 0x7f33a406ccd0 2023-11-25T05:26:23.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.526+0000 7f33a17fa640 1 -- 172.21.15.7:0/939092817 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f339804e450 con 0x7f33a406ccd0 2023-11-25T05:26:23.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.526+0000 7f33a17fa640 1 --2- 172.21.15.7:0/939092817 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f339006c7d0 0x7f339006ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.526+0000 7f33a8eb9640 1 --2- 172.21.15.7:0/939092817 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f339006c7d0 0x7f339006ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.526+0000 7f33a8eb9640 10 monclient: get_auth_request con 0x7f339006c7d0 auth_method 0 2023-11-25T05:26:23.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.527+0000 7f33a17fa640 10 client.21265.objecter ms_handle_connect 0x7f339006c7d0 2023-11-25T05:26:23.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.527+0000 7f33a8eb9640 1 --2- 172.21.15.7:0/939092817 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f339006c7d0 0x7f339006ec90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f339c0012e0 tx=0x7f339c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:23.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.528+0000 7f33a17fa640 1 -- 172.21.15.7:0/939092817 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f339802e070 con 0x7f33a406ccd0 2023-11-25T05:26:23.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.528+0000 7f33a17fa640 10 client.21265.objecter ms_dispatch 0x7f33a4000b90 osd_map(918..918 src has 501..918) v4 2023-11-25T05:26:23.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.528+0000 7f33a17fa640 3 client.21265.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:23.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.528+0000 7f33a17fa640 3 client.21265.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:23.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.529+0000 7f33a17fa640 20 client.21265.objecter dump_active .. 0 homeless 2023-11-25T05:26:23.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.538+0000 7f33a17fa640 1 -- 172.21.15.7:0/939092817 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3398076ab0 con 0x7f33a406ccd0 2023-11-25T05:26:23.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.538+0000 7f33a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:23.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.538+0000 7f33a17fa640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:26:23.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.720+0000 7f33ab11d640 1 -- 172.21.15.7:0/939092817 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f33a414d6c0 con 0x7f339006c7d0 2023-11-25T05:26:23.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.720+0000 7f33a17fa640 1 -- 172.21.15.7:0/939092817 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f339c002040 con 0x7f339006c7d0 2023-11-25T05:26:23.722 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 3 to deep-scrub 2023-11-25T05:26:23.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 10 librados: watch_flush enter 2023-11-25T05:26:23.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 10 librados: watch_flush exit 2023-11-25T05:26:23.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 20 client.21265.objecter shutdown clearing up homeless session... 2023-11-25T05:26:23.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 10 client.21265.objecter successfully canceled tick 2023-11-25T05:26:23.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 1 -- 172.21.15.7:0/939092817 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f339006c7d0 msgr2=0x7f339006ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 1 --2- 172.21.15.7:0/939092817 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f339006c7d0 0x7f339006ec90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f339c0012e0 tx=0x7f339c009040 comp rx=0 tx=0).stop 2023-11-25T05:26:23.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 10 monclient: shutdown 2023-11-25T05:26:23.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:23.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 1 -- 172.21.15.7:0/939092817 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33a406ccd0 msgr2=0x7f33a4136d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 1 --2- 172.21.15.7:0/939092817 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33a406ccd0 0x7f33a4136d60 secure :-1 s=READY pgs=5159 cs=0 l=1 rev1=1 crypto rx=0x7f3398033000 tx=0x7f33980046c0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 1 -- 172.21.15.7:0/939092817 shutdown_connections 2023-11-25T05:26:23.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 1 --2- 172.21.15.7:0/939092817 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f339006c7d0 0x7f339006ec90 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 1 --2- 172.21.15.7:0/939092817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33a406d0b0 0x7f33a413fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 1 --2- 172.21.15.7:0/939092817 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33a406ccd0 0x7f33a4136d60 unknown :-1 s=CLOSED pgs=5159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 1 --2- 172.21.15.7:0/939092817 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33a406c0e0 0x7f33a40aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 1 -- 172.21.15.7:0/939092817 >> 172.21.15.7:0/939092817 conn(0x7f33a40b5ca0 msgr2=0x7f33a4073c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:23.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 1 -- 172.21.15.7:0/939092817 shutdown_connections 2023-11-25T05:26:23.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 1 -- 172.21.15.7:0/939092817 wait complete. 2023-11-25T05:26:23.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.722+0000 7f33ab11d640 1 librados: shutdown 2023-11-25T05:26:23.737 INFO:tasks.ceph:Scrubbing osd.4 2023-11-25T05:26:23.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-11-25T05:26:23.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.848+0000 7fb420990640 10 monclient: get_monmap_and_config 2023-11-25T05:26:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.848+0000 7fb420990640 10 monclient: build_initial_monmap 2023-11-25T05:26:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.848+0000 7fb420990640 10 monclient: monmap: 2023-11-25T05:26:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:23.849389+0000 2023-11-25T05:26:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:23.849389+0000 2023-11-25T05:26:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:23.851 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:23.851 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 1 Processor -- start 2023-11-25T05:26:23.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 1 -- start start 2023-11-25T05:26:23.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 10 monclient: init 2023-11-25T05:26:23.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:23.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-11-25T05:26:23.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb41c0afbf0 addr v1:172.21.15.7:6789/0 2023-11-25T05:26:23.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb41c06f7e0 0x7fb41c070410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 10 monclient(hunting): picked mon.noname-c con 0x7fb41c06f7e0 addr v2:172.21.15.90:3300/0 2023-11-25T05:26:23.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb41c13af50 0x7fb41c13b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 10 monclient(hunting): picked mon.noname-a con 0x7fb41c13af50 addr v2:172.21.15.7:3300/0 2023-11-25T05:26:23.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41c0717e0 con 0x7fb41c0afbf0 2023-11-25T05:26:23.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:23.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb41c0affd0 con 0x7fb41c13af50 2023-11-25T05:26:23.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:23.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb41c071ac0 con 0x7fb41c06f7e0 2023-11-25T05:26:23.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:23.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.849+0000 7fb420990640 10 monclient(hunting): authenticate will time out at 2710.092725s 2023-11-25T05:26:23.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.850+0000 7fb413fff640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb41c06f7e0 0x7fb41c070410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.850+0000 7fb41ad9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb41c13af50 0x7fb41c13b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.850+0000 7fb413fff640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb41c06f7e0 0x7fb41c070410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:39370/0 (socket says 172.21.15.7:39370) 2023-11-25T05:26:23.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.850+0000 7fb413fff640 1 -- 172.21.15.7:0/360086413 learned_addr learned my addr 172.21.15.7:0/360086413 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:23.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.850+0000 7fb413fff640 10 monclient(hunting): get_auth_request con 0x7fb41c06f7e0 auth_method 0 2023-11-25T05:26:23.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.850+0000 7fb413fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:23.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.850+0000 7fb413fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.850+0000 7fb413fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.851+0000 7fb41ad9d640 10 monclient(hunting): get_auth_request con 0x7fb41c13af50 auth_method 0 2023-11-25T05:26:23.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.851+0000 7fb41ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:23.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.851+0000 7fb41ad9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.851+0000 7fb41ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.851+0000 7fb413fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:23.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.851+0000 7fb413fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:23.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.851+0000 7fb413fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:23.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.851+0000 7fb41ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:23.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.851+0000 7fb41ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:23.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.851+0000 7fb41ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:23.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.852+0000 7fb413fff640 10 monclient(hunting): handle_auth_done global_id 21268 payload 274 2023-11-25T05:26:23.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.852+0000 7fb413fff640 1 -- 172.21.15.7:0/360086413 >> v2:172.21.15.7:3300/0 conn(0x7fb41c13af50 msgr2=0x7fb41c13b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:23.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.852+0000 7fb413fff640 1 --2- 172.21.15.7:0/360086413 >> v2:172.21.15.7:3300/0 conn(0x7fb41c13af50 0x7fb41c13b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.852+0000 7fb413fff640 1 -- 172.21.15.7:0/360086413 >> v1:172.21.15.7:6789/0 conn(0x7fb41c0afbf0 legacy=0x7fb41c071ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.852+0000 7fb413fff640 10 monclient: _finish_hunting 0 2023-11-25T05:26:23.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.852+0000 7fb413fff640 1 monclient: found mon.noname-c 2023-11-25T05:26:23.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.852+0000 7fb413fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:23.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.852+0000 7fb413fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:26:23.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.852+0000 7fb413fff640 1 -- 172.21.15.7:0/360086413 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c06fef0 con 0x7fb41c06f7e0 2023-11-25T05:26:23.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.854+0000 7fb413fff640 1 --2- 172.21.15.7:0/360086413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c06f7e0 0x7fb41c070410 secure :-1 s=READY pgs=5160 cs=0 l=1 rev1=1 crypto rx=0x7fb41c0b1280 tx=0x7fb40402cf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=328451bd50145065 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb419d9b640 1 -- 172.21.15.7:0/360086413 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40402dcc0 con 0x7fb41c06f7e0 2023-11-25T05:26:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb419d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:23.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb419d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:26:23.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb419d9b640 10 monclient: dump: 2023-11-25T05:26:23.871 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:23.871 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:23.871 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.871 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.872 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:23.872 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.872 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:23.872 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:23.872 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:23.872 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb419d9b640 10 monclient: _finish_auth 0 2023-11-25T05:26:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb419d9b640 10 monclient: _check_auth_tickets 2023-11-25T05:26:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb419d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb419d9b640 1 -- 172.21.15.7:0/360086413 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40402de20 con 0x7fb41c06f7e0 2023-11-25T05:26:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb419d9b640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb419d9b640 1 -- 172.21.15.7:0/360086413 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb404035660 con 0x7fb41c06f7e0 2023-11-25T05:26:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb420990640 5 monclient: authenticate success, global_id 21268 2023-11-25T05:26:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb420990640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb420990640 4 set_mon_vals no callback set 2023-11-25T05:26:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb419d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb419d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.857+0000 7fb419d9b640 10 monclient: dump: 2023-11-25T05:26:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:23.877 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:23.877 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:23.877 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.858+0000 7fb420990640 10 monclient: shutdown 2023-11-25T05:26:23.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.858+0000 7fb420990640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:23.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.858+0000 7fb420990640 1 -- 172.21.15.7:0/360086413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c06f7e0 msgr2=0x7fb41c070410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.858+0000 7fb420990640 1 --2- 172.21.15.7:0/360086413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c06f7e0 0x7fb41c070410 secure :-1 s=READY pgs=5160 cs=0 l=1 rev1=1 crypto rx=0x7fb41c0b1280 tx=0x7fb40402cf10 comp rx=0 tx=0).stop 2023-11-25T05:26:23.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.858+0000 7fb420990640 1 -- 172.21.15.7:0/360086413 shutdown_connections 2023-11-25T05:26:23.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.858+0000 7fb420990640 1 --2- 172.21.15.7:0/360086413 >> v2:172.21.15.7:3300/0 conn(0x7fb41c13af50 0x7fb41c13b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.858+0000 7fb420990640 1 --2- 172.21.15.7:0/360086413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c06f7e0 0x7fb41c070410 unknown :-1 s=CLOSED pgs=5160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.858+0000 7fb420990640 1 -- 172.21.15.7:0/360086413 >> 172.21.15.7:0/360086413 conn(0x7fb41c0b01e0 msgr2=0x7fb41c0b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:23.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.858+0000 7fb420990640 1 -- 172.21.15.7:0/360086413 shutdown_connections 2023-11-25T05:26:23.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.858+0000 7fb420990640 1 -- 172.21.15.7:0/360086413 wait complete. 2023-11-25T05:26:23.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.859+0000 7fb420990640 10 monclient: build_initial_monmap 2023-11-25T05:26:23.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.859+0000 7fb420990640 10 monclient: monmap: 2023-11-25T05:26:23.880 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:23.880 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:23.880 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:23.880 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:23.880 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:23.880 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.880 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:23.880 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:23.880 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:23.881 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.859+0000 7fb420990640 1 librados: starting msgr at 2023-11-25T05:26:23.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.859+0000 7fb420990640 1 librados: starting objecter 2023-11-25T05:26:23.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.859+0000 7fb420990640 1 Processor -- start 2023-11-25T05:26:23.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.859+0000 7fb420990640 1 -- start start 2023-11-25T05:26:23.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.859+0000 7fb420990640 1 librados: setting wanted keys 2023-11-25T05:26:23.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.859+0000 7fb420990640 1 librados: calling monclient init 2023-11-25T05:26:23.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.859+0000 7fb420990640 10 monclient: init 2023-11-25T05:26:23.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:23.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:26:23.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c06f7e0 0x7fb41c134ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 10 monclient(hunting): picked mon.noname-a con 0x7fb41c06f7e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:23.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb41c0afbf0 0x7fb41c1d4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 10 monclient(hunting): picked mon.noname-c con 0x7fb41c0afbf0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:23.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c13af50 0x7fb41c1d8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 10 monclient(hunting): picked mon.noname-b con 0x7fb41c13af50 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:23.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:23.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c1d8370 con 0x7fb41c06f7e0 2023-11-25T05:26:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c1d8580 con 0x7fb41c13af50 2023-11-25T05:26:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c1d8fe0 con 0x7fb41c0afbf0 2023-11-25T05:26:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb420990640 10 monclient(hunting): authenticate will time out at 2710.103363s 2023-11-25T05:26:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.860+0000 7fb41a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c06f7e0 0x7fb41c134ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb413fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb41c0afbf0 0x7fb41c1d4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb41a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c06f7e0 0x7fb41c134ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43800/0 (socket says 172.21.15.7:43800) 2023-11-25T05:26:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb41ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c13af50 0x7fb41c1d8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb41a59c640 1 -- 172.21.15.7:0/1720693310 learned_addr learned my addr 172.21.15.7:0/1720693310 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb41a59c640 10 monclient(hunting): get_auth_request con 0x7fb41c06f7e0 auth_method 0 2023-11-25T05:26:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb41a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb41a59c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb41a59c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb413fff640 10 monclient(hunting): get_auth_request con 0x7fb41c0afbf0 auth_method 0 2023-11-25T05:26:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb413fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb413fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb413fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb41ad9d640 10 monclient(hunting): get_auth_request con 0x7fb41c13af50 auth_method 0 2023-11-25T05:26:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb41ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb41ad9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb41ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb413fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb413fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb413fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb41a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb41a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.861+0000 7fb41a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.862+0000 7fb41ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.862+0000 7fb41ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.862+0000 7fb41ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.864+0000 7fb41ad9d640 10 monclient(hunting): handle_auth_done global_id 21271 payload 824 2023-11-25T05:26:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.864+0000 7fb41ad9d640 1 -- 172.21.15.7:0/1720693310 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb41c0afbf0 msgr2=0x7fb41c1d4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.864+0000 7fb41ad9d640 1 --2- 172.21.15.7:0/1720693310 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb41c0afbf0 0x7fb41c1d4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.864+0000 7fb41ad9d640 1 -- 172.21.15.7:0/1720693310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c06f7e0 msgr2=0x7fb41c134ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.864+0000 7fb41ad9d640 1 --2- 172.21.15.7:0/1720693310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb41c06f7e0 0x7fb41c134ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.864+0000 7fb41ad9d640 10 monclient: _finish_hunting 0 2023-11-25T05:26:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.864+0000 7fb41ad9d640 1 monclient: found mon.noname-b 2023-11-25T05:26:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.864+0000 7fb41ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:23.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.864+0000 7fb41ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:26:23.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.864+0000 7fb41ad9d640 1 -- 172.21.15.7:0/1720693310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40400a670 con 0x7fb41c13af50 2023-11-25T05:26:23.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.866+0000 7fb41ad9d640 1 --2- 172.21.15.7:0/1720693310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c13af50 0x7fb41c1d8aa0 secure :-1 s=READY pgs=5161 cs=0 l=1 rev1=1 crypto rx=0x7fb41400eba0 tx=0x7fb41400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:23.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.866+0000 7fb412ffd640 10 client.?.objecter ms_handle_connect 0x7fb41c13af50 2023-11-25T05:26:23.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.866+0000 7fb412ffd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:23.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb412ffd640 1 -- 172.21.15.7:0/1720693310 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41400cd20 con 0x7fb41c13af50 2023-11-25T05:26:23.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb412ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:23.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb412ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:23.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb412ffd640 10 monclient: dump: 2023-11-25T05:26:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:23.889 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb412ffd640 10 monclient: _finish_auth 0 2023-11-25T05:26:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb412ffd640 10 monclient: _check_auth_tickets 2023-11-25T05:26:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb412ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb412ffd640 1 -- 172.21.15.7:0/1720693310 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41400f040 con 0x7fb41c13af50 2023-11-25T05:26:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb420990640 5 monclient: authenticate success, global_id 21271 2023-11-25T05:26:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb420990640 10 monclient: _renew_subs 2023-11-25T05:26:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb420990640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb420990640 1 -- 172.21.15.7:0/1720693310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41c1d9200 con 0x7fb41c13af50 2023-11-25T05:26:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb412ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb412ffd640 1 -- 172.21.15.7:0/1720693310 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4140146c0 con 0x7fb41c13af50 2023-11-25T05:26:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb412ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb412ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb412ffd640 10 monclient: dump: 2023-11-25T05:26:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:23.891 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb420990640 10 client.21271.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb420990640 10 monclient: _renew_subs 2023-11-25T05:26:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb41959a640 4 set_mon_vals no callback set 2023-11-25T05:26:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb420990640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb420990640 1 -- 172.21.15.7:0/1720693310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41c147e20 con 0x7fb41c13af50 2023-11-25T05:26:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.870+0000 7fb420990640 1 librados: init done 2023-11-25T05:26:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.871+0000 7fb420990640 10 client.21271.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.871+0000 7fb420990640 15 client.21271.objecter _session_command_op_assign -1 1 2023-11-25T05:26:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.871+0000 7fb420990640 20 client.21271.objecter _get_session osd=-1 returning homeless 2023-11-25T05:26:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.871+0000 7fb420990640 10 client.21271.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.871+0000 7fb420990640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.871+0000 7fb420990640 1 -- 172.21.15.7:0/1720693310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb41c1d87a0 con 0x7fb41c13af50 2023-11-25T05:26:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.888+0000 7fb412ffd640 1 -- 172.21.15.7:0/1720693310 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb414014820 con 0x7fb41c13af50 2023-11-25T05:26:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.889+0000 7fb412ffd640 1 --2- 172.21.15.7:0/1720693310 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb3fc06c820 0x7fb3fc06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.889+0000 7fb41a59c640 1 --2- 172.21.15.7:0/1720693310 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb3fc06c820 0x7fb3fc06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.889+0000 7fb41a59c640 10 monclient: get_auth_request con 0x7fb3fc06c820 auth_method 0 2023-11-25T05:26:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.889+0000 7fb41a59c640 1 --2- 172.21.15.7:0/1720693310 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb3fc06c820 0x7fb3fc06ece0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb40c001840 tx=0x7fb40c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.889+0000 7fb412ffd640 10 client.21271.objecter ms_handle_connect 0x7fb3fc06c820 2023-11-25T05:26:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 1 -- 172.21.15.7:0/1720693310 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7fb4140599b0 con 0x7fb41c13af50 2023-11-25T05:26:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 10 client.21271.objecter ms_dispatch 0x7fb41c000b90 osd_map(918..918 src has 501..918) v4 2023-11-25T05:26:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 3 client.21271.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 3 client.21271.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 10 client.21271.objecter checking command 1 2023-11-25T05:26:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 1 --2- 172.21.15.7:0/1720693310 >> [v2:172.21.15.90:6820/1576115380,v1:172.21.15.90:6823/1576115380] conn(0x7fb3fc0740a0 0x7fb3fc076540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 20 client.21271.objecter _get_session s=0x7fb3fc0739d0 osd=4 3 2023-11-25T05:26:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 20 client.21271.objecter put_session s=0x7fb3fc0739d0 osd=4 3 2023-11-25T05:26:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 15 client.21271.objecter _session_command_op_remove -1 1 2023-11-25T05:26:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 20 client.21271.objecter _get_session s=0x7fb3fc0739d0 osd=4 3 2023-11-25T05:26:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 20 client.21271.objecter get_session s=0x7fb3fc0739d0 osd=4 3 2023-11-25T05:26:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 15 client.21271.objecter _session_command_op_assign 4 1 2023-11-25T05:26:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 20 client.21271.objecter put_session s=0x7fb3fc0739d0 osd=4 4 2023-11-25T05:26:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 10 client.21271.objecter _send_command 1 2023-11-25T05:26:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 1 -- 172.21.15.7:0/1720693310 --> [v2:172.21.15.90:6820/1576115380,v1:172.21.15.90:6823/1576115380] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3fc076c10 con 0x7fb3fc0740a0 2023-11-25T05:26:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.891+0000 7fb412ffd640 20 client.21271.objecter dump_active .. 0 homeless 2023-11-25T05:26:23.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.892+0000 7fb413fff640 1 --2- 172.21.15.7:0/1720693310 >> [v2:172.21.15.90:6820/1576115380,v1:172.21.15.90:6823/1576115380] conn(0x7fb3fc0740a0 0x7fb3fc076540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:23.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.892+0000 7fb413fff640 10 monclient: get_auth_request con 0x7fb3fc0740a0 auth_method 0 2023-11-25T05:26:23.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.893+0000 7fb412ffd640 1 -- 172.21.15.7:0/1720693310 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=918) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb41400ce80 con 0x7fb41c13af50 2023-11-25T05:26:23.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.893+0000 7fb412ffd640 10 monclient: handle_get_version_reply finishing 1 version 918 2023-11-25T05:26:23.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.895+0000 7fb413fff640 1 --2- 172.21.15.7:0/1720693310 >> [v2:172.21.15.90:6820/1576115380,v1:172.21.15.90:6823/1576115380] conn(0x7fb3fc0740a0 0x7fb3fc076540 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:23.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.895+0000 7fb412ffd640 10 client.21271.objecter ms_handle_connect 0x7fb3fc0740a0 2023-11-25T05:26:23.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.918+0000 7fb412ffd640 1 -- 172.21.15.7:0/1720693310 <== osd.4 v2:172.21.15.90:6820/1576115380 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fb3fc076c10 con 0x7fb3fc0740a0 2023-11-25T05:26:23.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.918+0000 7fb412ffd640 10 client.21271.objecter ms_dispatch 0x7fb41c000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:26:23.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.918+0000 7fb412ffd640 10 client.21271.objecter _finish_command 1 = system:0 2023-11-25T05:26:23.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.918+0000 7fb412ffd640 20 client.21271.objecter put_session s=0x7fb3fc0739d0 osd=4 4 2023-11-25T05:26:23.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.918+0000 7fb412ffd640 15 client.21271.objecter _session_command_op_remove 4 1 2023-11-25T05:26:23.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.931+0000 7fb420990640 10 client.21271.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-11-25T05:26:23.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.931+0000 7fb420990640 15 client.21271.objecter _session_command_op_assign -1 2 2023-11-25T05:26:23.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.931+0000 7fb420990640 20 client.21271.objecter _get_session s=0x7fb3fc0739d0 osd=4 3 2023-11-25T05:26:23.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.931+0000 7fb420990640 20 client.21271.objecter put_session s=0x7fb3fc0739d0 osd=4 3 2023-11-25T05:26:23.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.931+0000 7fb420990640 20 client.21271.objecter _get_session s=0x7fb3fc0739d0 osd=4 3 2023-11-25T05:26:23.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.931+0000 7fb420990640 15 client.21271.objecter _session_command_op_remove -1 2 2023-11-25T05:26:23.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.931+0000 7fb420990640 20 client.21271.objecter get_session s=0x7fb3fc0739d0 osd=4 3 2023-11-25T05:26:23.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.931+0000 7fb420990640 15 client.21271.objecter _session_command_op_assign 4 2 2023-11-25T05:26:23.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.931+0000 7fb420990640 20 client.21271.objecter put_session s=0x7fb3fc0739d0 osd=4 4 2023-11-25T05:26:23.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.931+0000 7fb420990640 10 client.21271.objecter _send_command 2 2023-11-25T05:26:23.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.931+0000 7fb420990640 1 -- 172.21.15.7:0/1720693310 --> [v2:172.21.15.90:6820/1576115380,v1:172.21.15.90:6823/1576115380] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb41c075af0 con 0x7fb3fc0740a0 2023-11-25T05:26:23.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.945+0000 7fb412ffd640 1 -- 172.21.15.7:0/1720693310 <== osd.4 v2:172.21.15.90:6820/1576115380 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb41c075af0 con 0x7fb3fc0740a0 2023-11-25T05:26:23.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.945+0000 7fb412ffd640 10 client.21271.objecter ms_dispatch 0x7fb41c000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:26:23.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.945+0000 7fb412ffd640 10 client.21271.objecter _finish_command 2 = system:0 2023-11-25T05:26:23.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.945+0000 7fb412ffd640 20 client.21271.objecter put_session s=0x7fb3fc0739d0 osd=4 4 2023-11-25T05:26:23.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.945+0000 7fb412ffd640 15 client.21271.objecter _session_command_op_remove 4 2 2023-11-25T05:26:23.948 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-11-25T05:26:23.948 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-25T05:26:23.948 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-11-25T05:26:23.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.945+0000 7fb420990640 10 librados: watch_flush enter 2023-11-25T05:26:23.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.945+0000 7fb420990640 10 librados: watch_flush exit 2023-11-25T05:26:23.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.945+0000 7fb420990640 10 client.21271.objecter close_session for osd.4 2023-11-25T05:26:23.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.945+0000 7fb420990640 1 -- 172.21.15.7:0/1720693310 >> [v2:172.21.15.90:6820/1576115380,v1:172.21.15.90:6823/1576115380] conn(0x7fb3fc0740a0 msgr2=0x7fb3fc076540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.945+0000 7fb420990640 1 --2- 172.21.15.7:0/1720693310 >> [v2:172.21.15.90:6820/1576115380,v1:172.21.15.90:6823/1576115380] conn(0x7fb3fc0740a0 0x7fb3fc076540 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:23.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.945+0000 7fb420990640 20 client.21271.objecter put_session s=0x7fb3fc0739d0 osd=4 1 2023-11-25T05:26:23.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.945+0000 7fb420990640 20 client.21271.objecter shutdown clearing up homeless session... 2023-11-25T05:26:23.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.945+0000 7fb420990640 10 client.21271.objecter successfully canceled tick 2023-11-25T05:26:23.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.946+0000 7fb420990640 1 -- 172.21.15.7:0/1720693310 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb3fc06c820 msgr2=0x7fb3fc06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.946+0000 7fb420990640 1 --2- 172.21.15.7:0/1720693310 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fb3fc06c820 0x7fb3fc06ece0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb40c001840 tx=0x7fb40c009040 comp rx=0 tx=0).stop 2023-11-25T05:26:23.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.946+0000 7fb420990640 10 monclient: shutdown 2023-11-25T05:26:23.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.946+0000 7fb420990640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:23.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.946+0000 7fb420990640 1 -- 172.21.15.7:0/1720693310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c13af50 msgr2=0x7fb41c1d8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:23.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.946+0000 7fb420990640 1 --2- 172.21.15.7:0/1720693310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c13af50 0x7fb41c1d8aa0 secure :-1 s=READY pgs=5161 cs=0 l=1 rev1=1 crypto rx=0x7fb41400eba0 tx=0x7fb41400c490 comp rx=0 tx=0).stop 2023-11-25T05:26:23.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.946+0000 7fb41ad9d640 1 -- 172.21.15.7:0/1720693310 reap_dead start 2023-11-25T05:26:23.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.946+0000 7fb420990640 1 -- 172.21.15.7:0/1720693310 shutdown_connections 2023-11-25T05:26:23.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.946+0000 7fb420990640 1 -- 172.21.15.7:0/1720693310 >> 172.21.15.7:0/1720693310 conn(0x7fb41c0b01e0 msgr2=0x7fb41c0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:23.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.946+0000 7fb420990640 1 -- 172.21.15.7:0/1720693310 shutdown_connections 2023-11-25T05:26:23.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.946+0000 7fb420990640 1 -- 172.21.15.7:0/1720693310 wait complete. 2023-11-25T05:26:23.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:23.946+0000 7fb420990640 1 librados: shutdown 2023-11-25T05:26:23.958 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-11-25T05:26:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.069+0000 7f39483b0640 10 monclient: get_monmap_and_config 2023-11-25T05:26:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.069+0000 7f39483b0640 10 monclient: build_initial_monmap 2023-11-25T05:26:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.069+0000 7f39483b0640 10 monclient: monmap: 2023-11-25T05:26:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:24.070061+0000 2023-11-25T05:26:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:24.070061+0000 2023-11-25T05:26:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:24.071 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.069+0000 7f39483b0640 1 Processor -- start 2023-11-25T05:26:24.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.069+0000 7f39483b0640 1 -- start start 2023-11-25T05:26:24.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.069+0000 7f39483b0640 10 monclient: init 2023-11-25T05:26:24.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.070+0000 7f39483b0640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:24.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.070+0000 7f39483b0640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2023-11-25T05:26:24.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.070+0000 7f39483b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39400a8910 addr v1:172.21.15.7:6789/0 2023-11-25T05:26:24.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.070+0000 7f39483b0640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f39400a9b70 0x7f394013f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.070+0000 7f39483b0640 10 monclient(hunting): picked mon.noname-a con 0x7f39400a9b70 addr v2:172.21.15.7:3300/0 2023-11-25T05:26:24.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.070+0000 7f39483b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3940141f90 addr v1:172.21.15.90:6789/0 2023-11-25T05:26:24.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.070+0000 7f39483b0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39400a8cf0 con 0x7f39400a8910 2023-11-25T05:26:24.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.070+0000 7f39483b0640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39400aabd0 con 0x7f3940141f90 2023-11-25T05:26:24.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.070+0000 7f39483b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:24.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.070+0000 7f39483b0640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39400aaeb0 con 0x7f39400a9b70 2023-11-25T05:26:24.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.070+0000 7f39483b0640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:24.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.070+0000 7f39483b0640 10 monclient(hunting): authenticate will time out at 2710.313480s 2023-11-25T05:26:24.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.071+0000 7f394594b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f39400a9b70 0x7f394013f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:24.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.071+0000 7f394594b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f39400a9b70 0x7f394013f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43806/0 (socket says 172.21.15.7:43806) 2023-11-25T05:26:24.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.071+0000 7f394594b640 1 -- 172.21.15.7:0/83914133 learned_addr learned my addr 172.21.15.7:0/83914133 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:24.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.071+0000 7f394594b640 10 monclient(hunting): get_auth_request con 0x7f39400a9b70 auth_method 0 2023-11-25T05:26:24.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.071+0000 7f394594b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:24.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.071+0000 7f394594b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:24.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.071+0000 7f394594b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:24.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.072+0000 7f394594b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:24.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.072+0000 7f394594b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:24.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.072+0000 7f394594b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:24.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.074+0000 7f394594b640 10 monclient(hunting): handle_auth_done global_id 15951 payload 274 2023-11-25T05:26:24.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.074+0000 7f394594b640 1 -- 172.21.15.7:0/83914133 >> v1:172.21.15.90:6789/0 conn(0x7f3940141f90 legacy=0x7f39401423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:24.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.074+0000 7f394594b640 1 -- 172.21.15.7:0/83914133 >> v1:172.21.15.7:6789/0 conn(0x7f39400a8910 legacy=0x7f394013ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.074+0000 7f394594b640 10 monclient: _finish_hunting 0 2023-11-25T05:26:24.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.074+0000 7f394594b640 1 monclient: found mon.noname-a 2023-11-25T05:26:24.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.074+0000 7f394594b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:24.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.074+0000 7f394594b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:26:24.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.074+0000 7f394594b640 1 -- 172.21.15.7:0/83914133 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39400a9720 con 0x7f39400a9b70 2023-11-25T05:26:24.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.078+0000 7f394594b640 1 --2- 172.21.15.7:0/83914133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39400a9b70 0x7f394013f370 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f3930009870 tx=0x7f393002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d1de78a7834899 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:24.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f394514a640 1 -- 172.21.15.7:0/83914133 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930030860 con 0x7f39400a9b70 2023-11-25T05:26:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f394514a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f394514a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:26:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f394514a640 10 monclient: dump: 2023-11-25T05:26:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:24.089 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.090 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.091 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:24.091 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.092 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:24.092 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:24.093 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:24.093 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f394514a640 10 monclient: _finish_auth 0 2023-11-25T05:26:24.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f394514a640 10 monclient: _check_auth_tickets 2023-11-25T05:26:24.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f394514a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:24.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f394514a640 1 -- 172.21.15.7:0/83914133 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39300309c0 con 0x7f39400a9b70 2023-11-25T05:26:24.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f394514a640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:24.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f394514a640 1 -- 172.21.15.7:0/83914133 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930030d50 con 0x7f39400a9b70 2023-11-25T05:26:24.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f39483b0640 5 monclient: authenticate success, global_id 15951 2023-11-25T05:26:24.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f39483b0640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:24.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f39483b0640 4 set_mon_vals no callback set 2023-11-25T05:26:24.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f394514a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:24.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f394514a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:26:24.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.082+0000 7f394514a640 10 monclient: dump: 2023-11-25T05:26:24.097 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:24.097 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:24.097 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.097 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.097 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:24.097 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.097 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:24.097 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:24.098 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:24.098 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.083+0000 7f39483b0640 10 monclient: shutdown 2023-11-25T05:26:24.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.083+0000 7f39483b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:24.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.083+0000 7f39483b0640 1 -- 172.21.15.7:0/83914133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39400a9b70 msgr2=0x7f394013f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.083+0000 7f39483b0640 1 --2- 172.21.15.7:0/83914133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39400a9b70 0x7f394013f370 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f3930009870 tx=0x7f393002b9a0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.083+0000 7f39483b0640 1 -- 172.21.15.7:0/83914133 shutdown_connections 2023-11-25T05:26:24.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.083+0000 7f39483b0640 1 --2- 172.21.15.7:0/83914133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39400a9b70 0x7f394013f370 unknown :-1 s=CLOSED pgs=3153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.083+0000 7f39483b0640 1 -- 172.21.15.7:0/83914133 >> 172.21.15.7:0/83914133 conn(0x7f39400b7780 msgr2=0x7f39400b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:24.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.083+0000 7f39483b0640 1 -- 172.21.15.7:0/83914133 shutdown_connections 2023-11-25T05:26:24.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.083+0000 7f39483b0640 1 -- 172.21.15.7:0/83914133 wait complete. 2023-11-25T05:26:24.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.084+0000 7f39483b0640 10 monclient: build_initial_monmap 2023-11-25T05:26:24.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.084+0000 7f39483b0640 10 monclient: monmap: 2023-11-25T05:26:24.100 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:24.100 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:24.100 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:24.100 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:24.100 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:24.100 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.100 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:24.100 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:24.100 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:24.100 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.084+0000 7f39483b0640 1 librados: starting msgr at 2023-11-25T05:26:24.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.084+0000 7f39483b0640 1 librados: starting objecter 2023-11-25T05:26:24.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.084+0000 7f39483b0640 1 Processor -- start 2023-11-25T05:26:24.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.084+0000 7f39483b0640 1 -- start start 2023-11-25T05:26:24.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.084+0000 7f39483b0640 1 librados: setting wanted keys 2023-11-25T05:26:24.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.084+0000 7f39483b0640 1 librados: calling monclient init 2023-11-25T05:26:24.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.084+0000 7f39483b0640 10 monclient: init 2023-11-25T05:26:24.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:24.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-25T05:26:24.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39400a8910 0x7f394013b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 10 monclient(hunting): picked mon.noname-c con 0x7f39400a8910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:24.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3940141f90 0x7f39400ac580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 10 monclient(hunting): picked mon.noname-a con 0x7f3940141f90 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:24.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3940139050 0x7f3940137d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 10 monclient(hunting): picked mon.noname-b con 0x7f3940139050 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:24.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:24.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3940139990 con 0x7f3940141f90 2023-11-25T05:26:24.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:24.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394013b770 con 0x7f3940139050 2023-11-25T05:26:24.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:24.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39400ac160 con 0x7f39400a8910 2023-11-25T05:26:24.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:24.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.085+0000 7f39483b0640 10 monclient(hunting): authenticate will time out at 2710.328366s 2023-11-25T05:26:24.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.086+0000 7f394594b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3940141f90 0x7f39400ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:24.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.086+0000 7f394594b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3940141f90 0x7f39400ac580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43810/0 (socket says 172.21.15.7:43810) 2023-11-25T05:26:24.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.086+0000 7f394594b640 1 -- 172.21.15.7:0/2128094517 learned_addr learned my addr 172.21.15.7:0/2128094517 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:24.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.086+0000 7f394594b640 10 monclient(hunting): get_auth_request con 0x7f3940141f90 auth_method 0 2023-11-25T05:26:24.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.086+0000 7f394594b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:24.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.086+0000 7f394594b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:24.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.086+0000 7f394594b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:24.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.086+0000 7f394594b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:24.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.086+0000 7f394594b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:24.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.086+0000 7f394594b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:24.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.089+0000 7f394594b640 10 monclient(hunting): handle_auth_done global_id 15954 payload 824 2023-11-25T05:26:24.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.089+0000 7f394594b640 1 -- 172.21.15.7:0/2128094517 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39400a8910 msgr2=0x7f394013b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.089+0000 7f394594b640 1 --2- 172.21.15.7:0/2128094517 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39400a8910 0x7f394013b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.089+0000 7f394594b640 1 -- 172.21.15.7:0/2128094517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3940139050 msgr2=0x7f3940137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.089+0000 7f394594b640 1 --2- 172.21.15.7:0/2128094517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3940139050 0x7f3940137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.089+0000 7f394594b640 10 monclient: _finish_hunting 0 2023-11-25T05:26:24.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.089+0000 7f394594b640 1 monclient: found mon.noname-a 2023-11-25T05:26:24.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.089+0000 7f394594b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:24.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.089+0000 7f394594b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:26:24.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.089+0000 7f394594b640 1 -- 172.21.15.7:0/2128094517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39300094d0 con 0x7f3940141f90 2023-11-25T05:26:24.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.092+0000 7f394594b640 1 --2- 172.21.15.7:0/2128094517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3940141f90 0x7f39400ac580 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 crypto rx=0x7f3930009870 tx=0x7f3930004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:24.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.092+0000 7f3936ffd640 10 client.?.objecter ms_handle_connect 0x7f3940141f90 2023-11-25T05:26:24.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.092+0000 7f3936ffd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:24.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.095+0000 7f3936ffd640 1 -- 172.21.15.7:0/2128094517 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930030810 con 0x7f3940141f90 2023-11-25T05:26:24.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f3936ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:24.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f3936ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:26:24.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f3936ffd640 10 monclient: dump: 2023-11-25T05:26:24.109 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:24.109 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:24.109 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.110 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.110 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:24.110 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.110 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:24.110 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:24.110 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:24.110 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f3936ffd640 10 monclient: _finish_auth 0 2023-11-25T05:26:24.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f3936ffd640 10 monclient: _check_auth_tickets 2023-11-25T05:26:24.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f3936ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:24.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f3936ffd640 1 -- 172.21.15.7:0/2128094517 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3930043400 con 0x7f3940141f90 2023-11-25T05:26:24.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f39483b0640 5 monclient: authenticate success, global_id 15954 2023-11-25T05:26:24.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f39483b0640 10 monclient: _renew_subs 2023-11-25T05:26:24.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f39483b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:26:24.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f39483b0640 1 -- 172.21.15.7:0/2128094517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394013bb30 con 0x7f3940141f90 2023-11-25T05:26:24.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f3936ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:24.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f3936ffd640 1 -- 172.21.15.7:0/2128094517 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39300437b0 con 0x7f3940141f90 2023-11-25T05:26:24.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f3936ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:24.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f3936ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:26:24.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f3936ffd640 10 monclient: dump: 2023-11-25T05:26:24.113 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:24.113 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:24.113 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.113 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.113 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:24.113 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.114 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:24.114 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:24.114 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:24.114 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f3944949640 4 set_mon_vals no callback set 2023-11-25T05:26:24.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f39483b0640 10 client.15954.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:24.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f39483b0640 10 monclient: _renew_subs 2023-11-25T05:26:24.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f39483b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:26:24.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f39483b0640 1 -- 172.21.15.7:0/2128094517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39400accf0 con 0x7f3940141f90 2023-11-25T05:26:24.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.096+0000 7f39483b0640 1 librados: init done 2023-11-25T05:26:24.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.097+0000 7f39483b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:24.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.097+0000 7f39483b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:24.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.097+0000 7f39483b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:26:24.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.097+0000 7f39483b0640 1 -- 172.21.15.7:0/2128094517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3940075b60 con 0x7f3940141f90 2023-11-25T05:26:24.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.119+0000 7f3936ffd640 1 -- 172.21.15.7:0/2128094517 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3930043910 con 0x7f3940141f90 2023-11-25T05:26:24.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.120+0000 7f3936ffd640 1 --2- 172.21.15.7:0/2128094517 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f392406c820 0x7f392406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.120+0000 7f394614c640 1 --2- 172.21.15.7:0/2128094517 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f392406c820 0x7f392406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:24.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.120+0000 7f394614c640 10 monclient: get_auth_request con 0x7f392406c820 auth_method 0 2023-11-25T05:26:24.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.120+0000 7f394614c640 1 --2- 172.21.15.7:0/2128094517 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f392406c820 0x7f392406ece0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f393c0012e0 tx=0x7f393c007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:24.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.120+0000 7f3936ffd640 10 client.15954.objecter ms_handle_connect 0x7f392406c820 2023-11-25T05:26:24.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.378+0000 7f3936ffd640 1 -- 172.21.15.7:0/2128094517 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f393003e070 con 0x7f3940141f90 2023-11-25T05:26:24.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.378+0000 7f3936ffd640 10 client.15954.objecter ms_dispatch 0x7f3940000b90 osd_map(918..918 src has 501..918) v4 2023-11-25T05:26:24.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.378+0000 7f3936ffd640 3 client.15954.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:24.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.378+0000 7f3936ffd640 3 client.15954.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:24.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.378+0000 7f3936ffd640 20 client.15954.objecter dump_active .. 0 homeless 2023-11-25T05:26:24.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.378+0000 7f3936ffd640 1 -- 172.21.15.7:0/2128094517 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3930078510 con 0x7f3940141f90 2023-11-25T05:26:24.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.378+0000 7f3936ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:24.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.378+0000 7f3936ffd640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:26:24.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.560+0000 7f39483b0640 1 -- 172.21.15.7:0/2128094517 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f394014d670 con 0x7f392406c820 2023-11-25T05:26:24.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.560+0000 7f3936ffd640 1 -- 172.21.15.7:0/2128094517 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f393c001920 con 0x7f392406c820 2023-11-25T05:26:24.562 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 4 to deep-scrub 2023-11-25T05:26:24.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 10 librados: watch_flush enter 2023-11-25T05:26:24.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 10 librados: watch_flush exit 2023-11-25T05:26:24.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 20 client.15954.objecter shutdown clearing up homeless session... 2023-11-25T05:26:24.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 10 client.15954.objecter successfully canceled tick 2023-11-25T05:26:24.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 1 -- 172.21.15.7:0/2128094517 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f392406c820 msgr2=0x7f392406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 1 --2- 172.21.15.7:0/2128094517 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f392406c820 0x7f392406ece0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f393c0012e0 tx=0x7f393c007d30 comp rx=0 tx=0).stop 2023-11-25T05:26:24.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 10 monclient: shutdown 2023-11-25T05:26:24.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:24.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 1 -- 172.21.15.7:0/2128094517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3940141f90 msgr2=0x7f39400ac580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 1 --2- 172.21.15.7:0/2128094517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3940141f90 0x7f39400ac580 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 crypto rx=0x7f3930009870 tx=0x7f3930004730 comp rx=0 tx=0).stop 2023-11-25T05:26:24.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 1 -- 172.21.15.7:0/2128094517 shutdown_connections 2023-11-25T05:26:24.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 1 --2- 172.21.15.7:0/2128094517 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f392406c820 0x7f392406ece0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 1 --2- 172.21.15.7:0/2128094517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3940139050 0x7f3940137d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 1 --2- 172.21.15.7:0/2128094517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3940141f90 0x7f39400ac580 unknown :-1 s=CLOSED pgs=3154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 1 --2- 172.21.15.7:0/2128094517 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39400a8910 0x7f394013b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 1 -- 172.21.15.7:0/2128094517 >> 172.21.15.7:0/2128094517 conn(0x7f39400b7780 msgr2=0x7f3940136950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:24.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 1 -- 172.21.15.7:0/2128094517 shutdown_connections 2023-11-25T05:26:24.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 1 -- 172.21.15.7:0/2128094517 wait complete. 2023-11-25T05:26:24.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.562+0000 7f39483b0640 1 librados: shutdown 2023-11-25T05:26:24.575 INFO:tasks.ceph:Scrubbing osd.5 2023-11-25T05:26:24.575 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-11-25T05:26:24.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.683+0000 7ffa8c50c640 10 monclient: get_monmap_and_config 2023-11-25T05:26:24.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.683+0000 7ffa8c50c640 10 monclient: build_initial_monmap 2023-11-25T05:26:24.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.683+0000 7ffa8c50c640 10 monclient: monmap: 2023-11-25T05:26:24.685 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:24.685 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:24.685 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:24.684648+0000 2023-11-25T05:26:24.685 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:24.684648+0000 2023-11-25T05:26:24.686 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:24.686 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.686 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:24.686 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:24.686 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:24.686 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:24.686 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:24.686 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.684+0000 7ffa8c50c640 1 Processor -- start 2023-11-25T05:26:24.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.684+0000 7ffa8c50c640 1 -- start start 2023-11-25T05:26:24.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.684+0000 7ffa8c50c640 10 monclient: init 2023-11-25T05:26:24.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8c50c640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:24.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8c50c640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-11-25T05:26:24.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8c50c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa840a88a0 0x7ffa8413f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8c50c640 10 monclient(hunting): picked mon.noname-b con 0x7ffa840a88a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:24.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8c50c640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ffa840a8c80 0x7ffa8413f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8c50c640 10 monclient(hunting): picked mon.noname-c con 0x7ffa840a8c80 addr v2:172.21.15.90:3300/0 2023-11-25T05:26:24.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8c50c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffa841420a0 addr v1:172.21.15.90:6789/0 2023-11-25T05:26:24.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8c50c640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa840b1a60 con 0x7ffa841420a0 2023-11-25T05:26:24.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8c50c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:24.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8c50c640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa840aa530 con 0x7ffa840a8c80 2023-11-25T05:26:24.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8c50c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:24.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8c50c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa84142c40 con 0x7ffa840a88a0 2023-11-25T05:26:24.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8c50c640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:24.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8c50c640 10 monclient(hunting): authenticate will time out at 2710.927976s 2023-11-25T05:26:24.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8a2a8640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa840a88a0 0x7ffa8413f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:24.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa89aa7640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ffa840a8c80 0x7ffa8413f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:24.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.685+0000 7ffa8a2a8640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa840a88a0 0x7ffa8413f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38384/0 (socket says 172.21.15.7:38384) 2023-11-25T05:26:24.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.686+0000 7ffa8a2a8640 1 -- 172.21.15.7:0/115111080 learned_addr learned my addr 172.21.15.7:0/115111080 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:24.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.686+0000 7ffa8a2a8640 10 monclient(hunting): get_auth_request con 0x7ffa840a88a0 auth_method 0 2023-11-25T05:26:24.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.686+0000 7ffa8a2a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:24.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.686+0000 7ffa8a2a8640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:24.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.686+0000 7ffa8a2a8640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:24.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.686+0000 7ffa89aa7640 10 monclient(hunting): get_auth_request con 0x7ffa840a8c80 auth_method 0 2023-11-25T05:26:24.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.686+0000 7ffa89aa7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:24.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.686+0000 7ffa89aa7640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:24.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.686+0000 7ffa89aa7640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:24.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.686+0000 7ffa8a2a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:24.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.686+0000 7ffa8a2a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:24.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.686+0000 7ffa8a2a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:24.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.687+0000 7ffa89aa7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:24.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.687+0000 7ffa89aa7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:24.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.687+0000 7ffa89aa7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:24.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.688+0000 7ffa8a2a8640 10 monclient(hunting): handle_auth_done global_id 17423 payload 274 2023-11-25T05:26:24.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.688+0000 7ffa8a2a8640 1 -- 172.21.15.7:0/115111080 >> v2:172.21.15.90:3300/0 conn(0x7ffa840a8c80 msgr2=0x7ffa8413f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:24.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.688+0000 7ffa8a2a8640 1 --2- 172.21.15.7:0/115111080 >> v2:172.21.15.90:3300/0 conn(0x7ffa840a8c80 0x7ffa8413f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.688+0000 7ffa8a2a8640 1 -- 172.21.15.7:0/115111080 >> v1:172.21.15.90:6789/0 conn(0x7ffa841420a0 legacy=0x7ffa84142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.688+0000 7ffa8a2a8640 10 monclient: _finish_hunting 0 2023-11-25T05:26:24.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.688+0000 7ffa8a2a8640 1 monclient: found mon.noname-b 2023-11-25T05:26:24.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.688+0000 7ffa8a2a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:24.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.688+0000 7ffa8a2a8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:26:24.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.688+0000 7ffa8a2a8640 1 -- 172.21.15.7:0/115111080 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa840aa810 con 0x7ffa840a88a0 2023-11-25T05:26:24.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.690+0000 7ffa8a2a8640 1 --2- 172.21.15.7:0/115111080 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa840a88a0 0x7ffa8413f030 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7ffa80009870 tx=0x7ffa8002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3ec9a79f573ca9a server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:24.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa892a6640 1 -- 172.21.15.7:0/115111080 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa80030840 con 0x7ffa840a88a0 2023-11-25T05:26:24.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa892a6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:24.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa892a6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:24.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa892a6640 10 monclient: dump: 2023-11-25T05:26:24.702 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:24.702 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:24.702 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.702 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.702 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:24.702 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.702 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:24.702 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:24.702 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:24.702 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa892a6640 10 monclient: _finish_auth 0 2023-11-25T05:26:24.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa892a6640 10 monclient: _check_auth_tickets 2023-11-25T05:26:24.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa892a6640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:24.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa892a6640 1 -- 172.21.15.7:0/115111080 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa800309a0 con 0x7ffa840a88a0 2023-11-25T05:26:24.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa8c50c640 5 monclient: authenticate success, global_id 17423 2023-11-25T05:26:24.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa8c50c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:26:24.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa892a6640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:24.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa8c50c640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:24.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa892a6640 1 -- 172.21.15.7:0/115111080 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa80030d30 con 0x7ffa840a88a0 2023-11-25T05:26:24.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa892a6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:24.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa8c50c640 4 set_mon_vals no callback set 2023-11-25T05:26:24.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa892a6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:26:24.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.693+0000 7ffa892a6640 10 monclient: dump: 2023-11-25T05:26:24.704 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:24.704 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:24.704 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.704 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.704 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:24.704 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.704 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:24.704 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:24.704 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:24.704 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.694+0000 7ffa8c50c640 10 monclient: shutdown 2023-11-25T05:26:24.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.694+0000 7ffa8c50c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:24.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.694+0000 7ffa8c50c640 1 -- 172.21.15.7:0/115111080 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa840a88a0 msgr2=0x7ffa8413f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.694+0000 7ffa8c50c640 1 --2- 172.21.15.7:0/115111080 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa840a88a0 0x7ffa8413f030 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7ffa80009870 tx=0x7ffa8002b980 comp rx=0 tx=0).stop 2023-11-25T05:26:24.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.694+0000 7ffa8c50c640 1 -- 172.21.15.7:0/115111080 shutdown_connections 2023-11-25T05:26:24.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.694+0000 7ffa8c50c640 1 --2- 172.21.15.7:0/115111080 >> v2:172.21.15.90:3300/0 conn(0x7ffa840a8c80 0x7ffa8413f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.694+0000 7ffa8c50c640 1 --2- 172.21.15.7:0/115111080 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa840a88a0 0x7ffa8413f030 unknown :-1 s=CLOSED pgs=3072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.694+0000 7ffa8c50c640 1 -- 172.21.15.7:0/115111080 >> 172.21.15.7:0/115111080 conn(0x7ffa84061890 msgr2=0x7ffa840b7650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:24.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.694+0000 7ffa8c50c640 1 -- 172.21.15.7:0/115111080 shutdown_connections 2023-11-25T05:26:24.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.694+0000 7ffa8c50c640 1 -- 172.21.15.7:0/115111080 wait complete. 2023-11-25T05:26:24.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.695+0000 7ffa8c50c640 10 monclient: build_initial_monmap 2023-11-25T05:26:24.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.695+0000 7ffa8c50c640 10 monclient: monmap: 2023-11-25T05:26:24.705 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:24.705 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:24.705 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:24.706 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:24.706 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:24.706 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.706 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:24.706 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:24.706 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:24.706 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.695+0000 7ffa8c50c640 1 librados: starting msgr at 2023-11-25T05:26:24.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.695+0000 7ffa8c50c640 1 librados: starting objecter 2023-11-25T05:26:24.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.695+0000 7ffa8c50c640 1 Processor -- start 2023-11-25T05:26:24.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.695+0000 7ffa8c50c640 1 -- start start 2023-11-25T05:26:24.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.695+0000 7ffa8c50c640 1 librados: setting wanted keys 2023-11-25T05:26:24.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.695+0000 7ffa8c50c640 1 librados: calling monclient init 2023-11-25T05:26:24.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.695+0000 7ffa8c50c640 10 monclient: init 2023-11-25T05:26:24.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:24.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-25T05:26:24.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa840a88a0 0x7ffa841364e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 10 monclient(hunting): picked mon.noname-c con 0x7ffa840a88a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:24.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa840a8c80 0x7ffa84134b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 10 monclient(hunting): picked mon.noname-b con 0x7ffa840a8c80 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:24.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa841420a0 0x7ffa8413cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 10 monclient(hunting): picked mon.noname-a con 0x7ffa841420a0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:24.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:24.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa84135ab0 con 0x7ffa841420a0 2023-11-25T05:26:24.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:24.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa84135d00 con 0x7ffa840a8c80 2023-11-25T05:26:24.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:24.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8413e770 con 0x7ffa840a88a0 2023-11-25T05:26:24.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:24.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8c50c640 10 monclient(hunting): authenticate will time out at 2710.939157s 2023-11-25T05:26:24.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8a2a8640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa840a88a0 0x7ffa841364e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:24.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa8aaa9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa841420a0 0x7ffa8413cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:24.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.696+0000 7ffa89aa7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa840a8c80 0x7ffa84134b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:24.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8a2a8640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa840a88a0 0x7ffa841364e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38398/0 (socket says 172.21.15.7:38398) 2023-11-25T05:26:24.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8a2a8640 1 -- 172.21.15.7:0/2534906031 learned_addr learned my addr 172.21.15.7:0/2534906031 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:24.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8a2a8640 10 monclient(hunting): get_auth_request con 0x7ffa840a88a0 auth_method 0 2023-11-25T05:26:24.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8a2a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:24.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8a2a8640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:24.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8a2a8640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa89aa7640 10 monclient(hunting): get_auth_request con 0x7ffa840a8c80 auth_method 0 2023-11-25T05:26:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa89aa7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa89aa7640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa89aa7640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8aaa9640 10 monclient(hunting): get_auth_request con 0x7ffa841420a0 auth_method 0 2023-11-25T05:26:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8aaa9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8aaa9640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8aaa9640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8a2a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8a2a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8a2a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa89aa7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa89aa7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa89aa7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8aaa9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8aaa9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.697+0000 7ffa8aaa9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.700+0000 7ffa89aa7640 10 monclient(hunting): handle_auth_done global_id 21280 payload 824 2023-11-25T05:26:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.700+0000 7ffa89aa7640 1 -- 172.21.15.7:0/2534906031 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa840a88a0 msgr2=0x7ffa841364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.700+0000 7ffa89aa7640 1 --2- 172.21.15.7:0/2534906031 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa840a88a0 0x7ffa841364e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.700+0000 7ffa89aa7640 1 -- 172.21.15.7:0/2534906031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa841420a0 msgr2=0x7ffa8413cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.700+0000 7ffa89aa7640 1 --2- 172.21.15.7:0/2534906031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa841420a0 0x7ffa8413cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.700+0000 7ffa89aa7640 10 monclient: _finish_hunting 0 2023-11-25T05:26:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.700+0000 7ffa89aa7640 1 monclient: found mon.noname-b 2023-11-25T05:26:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.700+0000 7ffa89aa7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:24.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.700+0000 7ffa89aa7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:26:24.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.700+0000 7ffa89aa7640 1 -- 172.21.15.7:0/2534906031 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa800094d0 con 0x7ffa840a8c80 2023-11-25T05:26:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.702+0000 7ffa89aa7640 1 --2- 172.21.15.7:0/2534906031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa840a8c80 0x7ffa84134b80 secure :-1 s=READY pgs=5163 cs=0 l=1 rev1=1 crypto rx=0x7ffa7400e850 tx=0x7ffa7400ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.702+0000 7ffa7affd640 10 client.?.objecter ms_handle_connect 0x7ffa840a8c80 2023-11-25T05:26:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.702+0000 7ffa7affd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.705+0000 7ffa7affd640 1 -- 172.21.15.7:0/2534906031 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa740095b0 con 0x7ffa840a8c80 2023-11-25T05:26:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa7affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa7affd640 10 monclient: dump: 2023-11-25T05:26:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.713 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.713 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:24.713 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.713 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:24.713 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:24.713 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:24.713 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa7affd640 10 monclient: _finish_auth 0 2023-11-25T05:26:24.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa7affd640 10 monclient: _check_auth_tickets 2023-11-25T05:26:24.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:24.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa7affd640 1 -- 172.21.15.7:0/2534906031 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7400c940 con 0x7ffa840a8c80 2023-11-25T05:26:24.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa7affd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:24.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa7affd640 1 -- 172.21.15.7:0/2534906031 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7400ccd0 con 0x7ffa840a8c80 2023-11-25T05:26:24.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:24.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:24.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa7affd640 10 monclient: dump: 2023-11-25T05:26:24.714 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:24.714 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:24.714 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.714 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.714 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:24.714 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.714 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:24.714 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:24.714 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:24.714 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa8c50c640 5 monclient: authenticate success, global_id 21280 2023-11-25T05:26:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa8c50c640 10 monclient: _renew_subs 2023-11-25T05:26:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa8c50c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa8c50c640 1 -- 172.21.15.7:0/2534906031 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa840b1f50 con 0x7ffa840a8c80 2023-11-25T05:26:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa88aa5640 4 set_mon_vals no callback set 2023-11-25T05:26:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa8c50c640 10 client.21280.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa8c50c640 10 monclient: _renew_subs 2023-11-25T05:26:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa8c50c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa8c50c640 1 -- 172.21.15.7:0/2534906031 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa840b2410 con 0x7ffa840a8c80 2023-11-25T05:26:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa8c50c640 1 librados: init done 2023-11-25T05:26:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa8c50c640 10 client.21280.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa8c50c640 15 client.21280.objecter _session_command_op_assign -1 1 2023-11-25T05:26:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa8c50c640 20 client.21280.objecter _get_session osd=-1 returning homeless 2023-11-25T05:26:24.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa8c50c640 10 client.21280.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:24.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa8c50c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:24.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.706+0000 7ffa8c50c640 1 -- 172.21.15.7:0/2534906031 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffa84069cf0 con 0x7ffa840a8c80 2023-11-25T05:26:24.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.726+0000 7ffa7affd640 1 -- 172.21.15.7:0/2534906031 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ffa74013470 con 0x7ffa840a8c80 2023-11-25T05:26:24.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.726+0000 7ffa7affd640 1 --2- 172.21.15.7:0/2534906031 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ffa6806c7d0 0x7ffa6806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.726+0000 7ffa8a2a8640 1 --2- 172.21.15.7:0/2534906031 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ffa6806c7d0 0x7ffa6806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:24.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.726+0000 7ffa8a2a8640 10 monclient: get_auth_request con 0x7ffa6806c7d0 auth_method 0 2023-11-25T05:26:24.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.727+0000 7ffa8a2a8640 1 --2- 172.21.15.7:0/2534906031 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ffa6806c7d0 0x7ffa6806ec90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ffa8002b9c0 tx=0x7ffa80008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:24.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.727+0000 7ffa7affd640 10 client.21280.objecter ms_handle_connect 0x7ffa6806c7d0 2023-11-25T05:26:24.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 1 -- 172.21.15.7:0/2534906031 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7ffa7400ce30 con 0x7ffa840a8c80 2023-11-25T05:26:24.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 10 client.21280.objecter ms_dispatch 0x7ffa84000b90 osd_map(918..918 src has 501..918) v4 2023-11-25T05:26:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 3 client.21280.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 3 client.21280.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 10 client.21280.objecter checking command 1 2023-11-25T05:26:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 1 --2- 172.21.15.7:0/2534906031 >> [v2:172.21.15.90:6810/132027349,v1:172.21.15.90:6811/132027349] conn(0x7ffa68074050 0x7ffa680764f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 20 client.21280.objecter _get_session s=0x7ffa68073980 osd=5 3 2023-11-25T05:26:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 20 client.21280.objecter put_session s=0x7ffa68073980 osd=5 3 2023-11-25T05:26:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 15 client.21280.objecter _session_command_op_remove -1 1 2023-11-25T05:26:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 20 client.21280.objecter _get_session s=0x7ffa68073980 osd=5 3 2023-11-25T05:26:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 20 client.21280.objecter get_session s=0x7ffa68073980 osd=5 3 2023-11-25T05:26:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 15 client.21280.objecter _session_command_op_assign 5 1 2023-11-25T05:26:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 20 client.21280.objecter put_session s=0x7ffa68073980 osd=5 4 2023-11-25T05:26:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 10 client.21280.objecter _send_command 1 2023-11-25T05:26:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 1 -- 172.21.15.7:0/2534906031 --> [v2:172.21.15.90:6810/132027349,v1:172.21.15.90:6811/132027349] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa68076bc0 con 0x7ffa68074050 2023-11-25T05:26:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.729+0000 7ffa7affd640 20 client.21280.objecter dump_active .. 0 homeless 2023-11-25T05:26:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.730+0000 7ffa8aaa9640 1 --2- 172.21.15.7:0/2534906031 >> [v2:172.21.15.90:6810/132027349,v1:172.21.15.90:6811/132027349] conn(0x7ffa68074050 0x7ffa680764f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.730+0000 7ffa7affd640 1 -- 172.21.15.7:0/2534906031 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=918) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa74054c10 con 0x7ffa840a8c80 2023-11-25T05:26:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.730+0000 7ffa7affd640 10 monclient: handle_get_version_reply finishing 1 version 918 2023-11-25T05:26:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.730+0000 7ffa8aaa9640 10 monclient: get_auth_request con 0x7ffa68074050 auth_method 0 2023-11-25T05:26:24.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.733+0000 7ffa7affd640 10 client.21280.objecter ms_handle_connect 0x7ffa68074050 2023-11-25T05:26:24.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.733+0000 7ffa8aaa9640 1 --2- 172.21.15.7:0/2534906031 >> [v2:172.21.15.90:6810/132027349,v1:172.21.15.90:6811/132027349] conn(0x7ffa68074050 0x7ffa680764f0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:24.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.758+0000 7ffa7affd640 1 -- 172.21.15.7:0/2534906031 <== osd.5 v2:172.21.15.90:6810/132027349 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ffa68076bc0 con 0x7ffa68074050 2023-11-25T05:26:24.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.758+0000 7ffa7affd640 10 client.21280.objecter ms_dispatch 0x7ffa84000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:26:24.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.758+0000 7ffa7affd640 10 client.21280.objecter _finish_command 1 = system:0 2023-11-25T05:26:24.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.758+0000 7ffa7affd640 20 client.21280.objecter put_session s=0x7ffa68073980 osd=5 4 2023-11-25T05:26:24.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.758+0000 7ffa7affd640 15 client.21280.objecter _session_command_op_remove 5 1 2023-11-25T05:26:24.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.771+0000 7ffa8c50c640 10 client.21280.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-11-25T05:26:24.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.771+0000 7ffa8c50c640 15 client.21280.objecter _session_command_op_assign -1 2 2023-11-25T05:26:24.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.771+0000 7ffa8c50c640 20 client.21280.objecter _get_session s=0x7ffa68073980 osd=5 3 2023-11-25T05:26:24.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.771+0000 7ffa8c50c640 20 client.21280.objecter put_session s=0x7ffa68073980 osd=5 3 2023-11-25T05:26:24.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.771+0000 7ffa8c50c640 20 client.21280.objecter _get_session s=0x7ffa68073980 osd=5 3 2023-11-25T05:26:24.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.771+0000 7ffa8c50c640 15 client.21280.objecter _session_command_op_remove -1 2 2023-11-25T05:26:24.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.771+0000 7ffa8c50c640 20 client.21280.objecter get_session s=0x7ffa68073980 osd=5 3 2023-11-25T05:26:24.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.771+0000 7ffa8c50c640 15 client.21280.objecter _session_command_op_assign 5 2 2023-11-25T05:26:24.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.771+0000 7ffa8c50c640 20 client.21280.objecter put_session s=0x7ffa68073980 osd=5 4 2023-11-25T05:26:24.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.771+0000 7ffa8c50c640 10 client.21280.objecter _send_command 2 2023-11-25T05:26:24.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.771+0000 7ffa8c50c640 1 -- 172.21.15.7:0/2534906031 --> [v2:172.21.15.90:6810/132027349,v1:172.21.15.90:6811/132027349] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ffa8413e930 con 0x7ffa68074050 2023-11-25T05:26:24.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa7affd640 1 -- 172.21.15.7:0/2534906031 <== osd.5 v2:172.21.15.90:6810/132027349 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ffa8413e930 con 0x7ffa68074050 2023-11-25T05:26:24.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa7affd640 10 client.21280.objecter ms_dispatch 0x7ffa84000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:26:24.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa7affd640 10 client.21280.objecter _finish_command 2 = system:0 2023-11-25T05:26:24.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa7affd640 20 client.21280.objecter put_session s=0x7ffa68073980 osd=5 4 2023-11-25T05:26:24.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa7affd640 15 client.21280.objecter _session_command_op_remove 5 2 2023-11-25T05:26:24.788 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-11-25T05:26:24.788 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-25T05:26:24.788 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-11-25T05:26:24.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 10 librados: watch_flush enter 2023-11-25T05:26:24.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 10 librados: watch_flush exit 2023-11-25T05:26:24.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 10 client.21280.objecter close_session for osd.5 2023-11-25T05:26:24.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 1 -- 172.21.15.7:0/2534906031 >> [v2:172.21.15.90:6810/132027349,v1:172.21.15.90:6811/132027349] conn(0x7ffa68074050 msgr2=0x7ffa680764f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 1 --2- 172.21.15.7:0/2534906031 >> [v2:172.21.15.90:6810/132027349,v1:172.21.15.90:6811/132027349] conn(0x7ffa68074050 0x7ffa680764f0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 20 client.21280.objecter put_session s=0x7ffa68073980 osd=5 1 2023-11-25T05:26:24.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 20 client.21280.objecter shutdown clearing up homeless session... 2023-11-25T05:26:24.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 10 client.21280.objecter successfully canceled tick 2023-11-25T05:26:24.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 1 -- 172.21.15.7:0/2534906031 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ffa6806c7d0 msgr2=0x7ffa6806ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 1 --2- 172.21.15.7:0/2534906031 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ffa6806c7d0 0x7ffa6806ec90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ffa8002b9c0 tx=0x7ffa80008030 comp rx=0 tx=0).stop 2023-11-25T05:26:24.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 10 monclient: shutdown 2023-11-25T05:26:24.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:24.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 1 -- 172.21.15.7:0/2534906031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa840a8c80 msgr2=0x7ffa84134b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 1 --2- 172.21.15.7:0/2534906031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa840a8c80 0x7ffa84134b80 secure :-1 s=READY pgs=5163 cs=0 l=1 rev1=1 crypto rx=0x7ffa7400e850 tx=0x7ffa7400ed10 comp rx=0 tx=0).stop 2023-11-25T05:26:24.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8aaa9640 1 -- 172.21.15.7:0/2534906031 reap_dead start 2023-11-25T05:26:24.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 1 -- 172.21.15.7:0/2534906031 shutdown_connections 2023-11-25T05:26:24.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 1 -- 172.21.15.7:0/2534906031 >> 172.21.15.7:0/2534906031 conn(0x7ffa84061890 msgr2=0x7ffa840b6700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:24.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 1 -- 172.21.15.7:0/2534906031 shutdown_connections 2023-11-25T05:26:24.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 1 -- 172.21.15.7:0/2534906031 wait complete. 2023-11-25T05:26:24.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.785+0000 7ffa8c50c640 1 librados: shutdown 2023-11-25T05:26:24.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-11-25T05:26:24.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.904+0000 7f022e73e640 10 monclient: get_monmap_and_config 2023-11-25T05:26:24.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.904+0000 7f022e73e640 10 monclient: build_initial_monmap 2023-11-25T05:26:24.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.904+0000 7f022e73e640 10 monclient: monmap: 2023-11-25T05:26:24.906 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:24.906 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:24.906 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:24.905619+0000 2023-11-25T05:26:24.906 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:24.905619+0000 2023-11-25T05:26:24.906 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:24.907 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.907 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:24.907 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:24.907 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:24.907 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:24.907 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:24.907 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.905+0000 7f022e73e640 1 Processor -- start 2023-11-25T05:26:24.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.905+0000 7f022e73e640 1 -- start start 2023-11-25T05:26:24.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.905+0000 7f022e73e640 10 monclient: init 2023-11-25T05:26:24.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.906+0000 7f022e73e640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:24.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.906+0000 7f022e73e640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2023-11-25T05:26:24.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.906+0000 7f022e73e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02280a8950 addr v1:172.21.15.90:6789/0 2023-11-25T05:26:24.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.906+0000 7f022e73e640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f02280a8d30 0x7f02281457b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.906+0000 7f022e73e640 10 monclient(hunting): picked mon.noname-c con 0x7f02280a8d30 addr v2:172.21.15.90:3300/0 2023-11-25T05:26:24.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.906+0000 7f022e73e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0228145cf0 addr v1:172.21.15.7:6789/0 2023-11-25T05:26:24.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.906+0000 7f022e73e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0228145320 con 0x7f0228145cf0 2023-11-25T05:26:24.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.906+0000 7f022e73e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02280b2260 con 0x7f02280a8950 2023-11-25T05:26:24.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.906+0000 7f022e73e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:24.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.906+0000 7f022e73e640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02280aa5e0 con 0x7f02280a8d30 2023-11-25T05:26:24.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.906+0000 7f022e73e640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:24.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.906+0000 7f022e73e640 10 monclient(hunting): authenticate will time out at 2711.149050s 2023-11-25T05:26:24.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.906+0000 7f02277fe640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f02280a8d30 0x7f02281457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:24.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.906+0000 7f022ccdb640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0228145cf0 0x7f022814c6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49946/0 (socket says 172.21.15.7:49946) 2023-11-25T05:26:24.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.906+0000 7f022ccdb640 1 -- 172.21.15.7:0/2709517718 learned_addr learned my addr 172.21.15.7:0/2709517718 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:26:24.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.907+0000 7f02277fe640 10 monclient(hunting): get_auth_request con 0x7f02280a8d30 auth_method 0 2023-11-25T05:26:24.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.907+0000 7f02277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:24.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.907+0000 7f02277fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:24.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.907+0000 7f02277fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:24.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.907+0000 7f02277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:24.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.907+0000 7f02277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:24.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.907+0000 7f02277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:24.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.909+0000 7f0226ffd640 1 -- 172.21.15.7:0/2709517718 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325859387 0 0) 0x7f0228145320 con 0x7f0228145cf0 2023-11-25T05:26:24.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.909+0000 7f0226ffd640 10 monclient(hunting): _negotiate 2023-11-25T05:26:24.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.909+0000 7f0226ffd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:24.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.909+0000 7f0226ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:24.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.909+0000 7f0226ffd640 10 monclient(hunting): my global_id is 15960 2023-11-25T05:26:24.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.909+0000 7f0226ffd640 1 -- 172.21.15.7:0/2709517718 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0214003520 con 0x7f0228145cf0 2023-11-25T05:26:24.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.909+0000 7f02277fe640 10 monclient(hunting): handle_auth_done global_id 21283 payload 274 2023-11-25T05:26:24.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.909+0000 7f02277fe640 1 -- 172.21.15.7:0/2709517718 >> v1:172.21.15.90:6789/0 conn(0x7f02280a8950 legacy=0x7f022813f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.909+0000 7f02277fe640 1 -- 172.21.15.7:0/2709517718 >> v1:172.21.15.7:6789/0 conn(0x7f0228145cf0 legacy=0x7f022814c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.909+0000 7f02277fe640 10 monclient: _finish_hunting 0 2023-11-25T05:26:24.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.909+0000 7f02277fe640 1 monclient: found mon.noname-c 2023-11-25T05:26:24.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.909+0000 7f02277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:24.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.909+0000 7f02277fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:26:24.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.909+0000 7f02277fe640 1 -- 172.21.15.7:0/2709517718 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02280aa8c0 con 0x7f02280a8d30 2023-11-25T05:26:24.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.914+0000 7f02277fe640 1 --2- 172.21.15.7:0/2709517718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02280a8d30 0x7f02281457b0 secure :-1 s=READY pgs=5165 cs=0 l=1 rev1=1 crypto rx=0x7f02180097a0 tx=0x7f021802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=792b14d3121dc286 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:24.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f0226ffd640 1 -- 172.21.15.7:0/2709517718 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218030610 con 0x7f02280a8d30 2023-11-25T05:26:24.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f0226ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:24.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f0226ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:26:24.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f0226ffd640 10 monclient: dump: 2023-11-25T05:26:24.919 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:24.919 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:24.919 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.920 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.920 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:24.920 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.920 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:24.920 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:24.920 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:24.921 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f0226ffd640 10 monclient: _finish_auth 0 2023-11-25T05:26:24.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f0226ffd640 10 monclient: _check_auth_tickets 2023-11-25T05:26:24.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f0226ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:24.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f0226ffd640 1 -- 172.21.15.7:0/2709517718 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0218030770 con 0x7f02280a8d30 2023-11-25T05:26:24.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f022e73e640 5 monclient: authenticate success, global_id 21283 2023-11-25T05:26:24.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f022e73e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:26:24.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f0226ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:24.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f022e73e640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:24.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f0226ffd640 1 -- 172.21.15.7:0/2709517718 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218030b00 con 0x7f02280a8d30 2023-11-25T05:26:24.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f0226ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:24.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f022e73e640 4 set_mon_vals no callback set 2023-11-25T05:26:24.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f0226ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:24.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f0226ffd640 10 monclient: dump: 2023-11-25T05:26:24.923 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:24.923 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:24.923 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.923 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.923 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:24.923 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.923 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:24.923 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:24.923 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:24.924 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f022e73e640 10 monclient: shutdown 2023-11-25T05:26:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f022e73e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f022e73e640 1 -- 172.21.15.7:0/2709517718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02280a8d30 msgr2=0x7f02281457b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f022e73e640 1 --2- 172.21.15.7:0/2709517718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02280a8d30 0x7f02281457b0 secure :-1 s=READY pgs=5165 cs=0 l=1 rev1=1 crypto rx=0x7f02180097a0 tx=0x7f021802b960 comp rx=0 tx=0).stop 2023-11-25T05:26:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f022e73e640 1 -- 172.21.15.7:0/2709517718 shutdown_connections 2023-11-25T05:26:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f022e73e640 1 --2- 172.21.15.7:0/2709517718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02280a8d30 0x7f02281457b0 unknown :-1 s=CLOSED pgs=5165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f022e73e640 1 -- 172.21.15.7:0/2709517718 >> 172.21.15.7:0/2709517718 conn(0x7f0228061970 msgr2=0x7f02280b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f022e73e640 1 -- 172.21.15.7:0/2709517718 shutdown_connections 2023-11-25T05:26:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.918+0000 7f022e73e640 1 -- 172.21.15.7:0/2709517718 wait complete. 2023-11-25T05:26:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.919+0000 7f022e73e640 10 monclient: build_initial_monmap 2023-11-25T05:26:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.919+0000 7f022e73e640 10 monclient: monmap: 2023-11-25T05:26:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:24.929 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.929 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:24.929 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:24.929 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:24.929 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.919+0000 7f022e73e640 1 librados: starting msgr at 2023-11-25T05:26:24.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.919+0000 7f022e73e640 1 librados: starting objecter 2023-11-25T05:26:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.919+0000 7f022e73e640 1 Processor -- start 2023-11-25T05:26:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 1 -- start start 2023-11-25T05:26:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 1 librados: setting wanted keys 2023-11-25T05:26:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 1 librados: calling monclient init 2023-11-25T05:26:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 10 monclient: init 2023-11-25T05:26:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:26:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02280a8950 0x7f0228144c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 10 monclient(hunting): picked mon.noname-a con 0x7f02280a8950 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02280a8d30 0x7f0228142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 10 monclient(hunting): picked mon.noname-c con 0x7f02280a8d30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0228145cf0 0x7f02280ab780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 10 monclient(hunting): picked mon.noname-b con 0x7f0228145cf0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02281d72d0 con 0x7f02280a8950 2023-11-25T05:26:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0228138760 con 0x7f0228145cf0 2023-11-25T05:26:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02281d6f30 con 0x7f02280a8d30 2023-11-25T05:26:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.920+0000 7f022e73e640 10 monclient(hunting): authenticate will time out at 2711.163454s 2023-11-25T05:26:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f022ccdb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0228145cf0 0x7f02280ab780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:24.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f0227fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02280a8950 0x7f0228144c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:24.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f02277fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02280a8d30 0x7f0228142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:24.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f022ccdb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0228145cf0 0x7f02280ab780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:39410/0 (socket says 172.21.15.7:39410) 2023-11-25T05:26:24.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f022ccdb640 1 -- 172.21.15.7:0/4227238561 learned_addr learned my addr 172.21.15.7:0/4227238561 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:24.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f022ccdb640 10 monclient(hunting): get_auth_request con 0x7f0228145cf0 auth_method 0 2023-11-25T05:26:24.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f022ccdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:24.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f022ccdb640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f022ccdb640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f02277fe640 10 monclient(hunting): get_auth_request con 0x7f02280a8d30 auth_method 0 2023-11-25T05:26:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f02277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f02277fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f02277fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f0227fff640 10 monclient(hunting): get_auth_request con 0x7f02280a8950 auth_method 0 2023-11-25T05:26:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f0227fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f0227fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f0227fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f022ccdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f022ccdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.921+0000 7f022ccdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.922+0000 7f0227fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.922+0000 7f0227fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.922+0000 7f0227fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.922+0000 7f02277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.922+0000 7f02277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.922+0000 7f02277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.924+0000 7f022ccdb640 10 monclient(hunting): handle_auth_done global_id 21289 payload 824 2023-11-25T05:26:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.924+0000 7f022ccdb640 1 -- 172.21.15.7:0/4227238561 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02280a8d30 msgr2=0x7f0228142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.924+0000 7f022ccdb640 1 --2- 172.21.15.7:0/4227238561 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02280a8d30 0x7f0228142140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.924+0000 7f022ccdb640 1 -- 172.21.15.7:0/4227238561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02280a8950 msgr2=0x7f0228144c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:24.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.924+0000 7f022ccdb640 1 --2- 172.21.15.7:0/4227238561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02280a8950 0x7f0228144c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:24.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.924+0000 7f022ccdb640 10 monclient: _finish_hunting 0 2023-11-25T05:26:24.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.924+0000 7f022ccdb640 1 monclient: found mon.noname-b 2023-11-25T05:26:24.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.924+0000 7f022ccdb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:24.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.924+0000 7f022ccdb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:26:24.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.924+0000 7f022ccdb640 1 -- 172.21.15.7:0/4227238561 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0218009400 con 0x7f0228145cf0 2023-11-25T05:26:24.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.926+0000 7f022ccdb640 1 --2- 172.21.15.7:0/4227238561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0228145cf0 0x7f02280ab780 secure :-1 s=READY pgs=5166 cs=0 l=1 rev1=1 crypto rx=0x7f022000cad0 tx=0x7f022000cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:24.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.926+0000 7f0224ff9640 10 client.?.objecter ms_handle_connect 0x7f0228145cf0 2023-11-25T05:26:24.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.926+0000 7f0224ff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:24.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f0224ff9640 1 -- 172.21.15.7:0/4227238561 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022000ac30 con 0x7f0228145cf0 2023-11-25T05:26:24.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f0224ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:24.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f0224ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:24.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f0224ff9640 10 monclient: dump: 2023-11-25T05:26:24.938 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:24.938 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:24.938 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.939 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.939 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:24.939 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.939 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:24.939 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:24.939 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:24.939 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f0224ff9640 10 monclient: _finish_auth 0 2023-11-25T05:26:24.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f0224ff9640 10 monclient: _check_auth_tickets 2023-11-25T05:26:24.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f0224ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:24.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f0224ff9640 1 -- 172.21.15.7:0/4227238561 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022000ad90 con 0x7f0228145cf0 2023-11-25T05:26:24.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f022e73e640 5 monclient: authenticate success, global_id 21289 2023-11-25T05:26:24.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f022e73e640 10 monclient: _renew_subs 2023-11-25T05:26:24.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f022e73e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:24.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f022e73e640 1 -- 172.21.15.7:0/4227238561 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0228143e30 con 0x7f0228145cf0 2023-11-25T05:26:24.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f0224ff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:24.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f0224ff9640 1 -- 172.21.15.7:0/4227238561 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0220012620 con 0x7f0228145cf0 2023-11-25T05:26:24.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f0224ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:24.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f0224ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:24.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f0224ff9640 10 monclient: dump: 2023-11-25T05:26:24.941 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:24.941 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:24.941 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.941 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:24.941 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:24.941 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:24.941 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:24.941 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:24.942 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:24.942 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:24.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f0225ffb640 4 set_mon_vals no callback set 2023-11-25T05:26:24.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f022e73e640 10 client.21289.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:24.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f022e73e640 10 monclient: _renew_subs 2023-11-25T05:26:24.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f022e73e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:24.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f022e73e640 1 -- 172.21.15.7:0/4227238561 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02281d78d0 con 0x7f0228145cf0 2023-11-25T05:26:24.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.930+0000 7f022e73e640 1 librados: init done 2023-11-25T05:26:24.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.931+0000 7f022e73e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:24.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.931+0000 7f022e73e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:24.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.931+0000 7f022e73e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:24.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:24.931+0000 7f022e73e640 1 -- 172.21.15.7:0/4227238561 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02281d7410 con 0x7f0228145cf0 2023-11-25T05:26:25.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.168+0000 7f0224ff9640 1 -- 172.21.15.7:0/4227238561 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0220007070 con 0x7f0228145cf0 2023-11-25T05:26:25.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.169+0000 7f0224ff9640 1 --2- 172.21.15.7:0/4227238561 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f020806c820 0x7f020806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:25.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.169+0000 7f0227fff640 1 --2- 172.21.15.7:0/4227238561 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f020806c820 0x7f020806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:25.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.169+0000 7f0227fff640 10 monclient: get_auth_request con 0x7f020806c820 auth_method 0 2023-11-25T05:26:25.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.169+0000 7f0227fff640 1 --2- 172.21.15.7:0/4227238561 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f020806c820 0x7f020806ece0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0210000f60 tx=0x7f0210009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:25.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.169+0000 7f0224ff9640 10 client.21289.objecter ms_handle_connect 0x7f020806c820 2023-11-25T05:26:25.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.170+0000 7f0224ff9640 1 -- 172.21.15.7:0/4227238561 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f0220058640 con 0x7f0228145cf0 2023-11-25T05:26:25.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.170+0000 7f0224ff9640 10 client.21289.objecter ms_dispatch 0x7f0228000b90 osd_map(918..918 src has 501..918) v4 2023-11-25T05:26:25.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.170+0000 7f0224ff9640 3 client.21289.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:25.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.170+0000 7f0224ff9640 3 client.21289.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:25.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.170+0000 7f0224ff9640 20 client.21289.objecter dump_active .. 0 homeless 2023-11-25T05:26:25.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.181+0000 7f0224ff9640 1 -- 172.21.15.7:0/4227238561 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0220054980 con 0x7f0228145cf0 2023-11-25T05:26:25.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.181+0000 7f0224ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:25.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.181+0000 7f0224ff9640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:26:25.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.359+0000 7f022e73e640 1 -- 172.21.15.7:0/4227238561 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f0228146160 con 0x7f020806c820 2023-11-25T05:26:25.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.359+0000 7f0224ff9640 1 -- 172.21.15.7:0/4227238561 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f02100014a0 con 0x7f020806c820 2023-11-25T05:26:25.361 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 5 to deep-scrub 2023-11-25T05:26:25.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 10 librados: watch_flush enter 2023-11-25T05:26:25.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 10 librados: watch_flush exit 2023-11-25T05:26:25.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 20 client.21289.objecter shutdown clearing up homeless session... 2023-11-25T05:26:25.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 10 client.21289.objecter successfully canceled tick 2023-11-25T05:26:25.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 1 -- 172.21.15.7:0/4227238561 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f020806c820 msgr2=0x7f020806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:25.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 1 --2- 172.21.15.7:0/4227238561 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f020806c820 0x7f020806ece0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0210000f60 tx=0x7f0210009040 comp rx=0 tx=0).stop 2023-11-25T05:26:25.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 10 monclient: shutdown 2023-11-25T05:26:25.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:25.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 1 -- 172.21.15.7:0/4227238561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0228145cf0 msgr2=0x7f02280ab780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:25.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 1 --2- 172.21.15.7:0/4227238561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0228145cf0 0x7f02280ab780 secure :-1 s=READY pgs=5166 cs=0 l=1 rev1=1 crypto rx=0x7f022000cad0 tx=0x7f022000cf90 comp rx=0 tx=0).stop 2023-11-25T05:26:25.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 1 -- 172.21.15.7:0/4227238561 shutdown_connections 2023-11-25T05:26:25.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 1 --2- 172.21.15.7:0/4227238561 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f020806c820 0x7f020806ece0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 1 --2- 172.21.15.7:0/4227238561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0228145cf0 0x7f02280ab780 unknown :-1 s=CLOSED pgs=5166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 1 --2- 172.21.15.7:0/4227238561 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02280a8d30 0x7f0228142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 1 --2- 172.21.15.7:0/4227238561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02280a8950 0x7f0228144c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 1 -- 172.21.15.7:0/4227238561 >> 172.21.15.7:0/4227238561 conn(0x7f0228061970 msgr2=0x7f02280b6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:25.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 1 -- 172.21.15.7:0/4227238561 shutdown_connections 2023-11-25T05:26:25.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 1 -- 172.21.15.7:0/4227238561 wait complete. 2023-11-25T05:26:25.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.361+0000 7f022e73e640 1 librados: shutdown 2023-11-25T05:26:25.374 INFO:tasks.ceph:Scrubbing osd.6 2023-11-25T05:26:25.374 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-11-25T05:26:25.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.483+0000 7f43a6e51640 10 monclient: get_monmap_and_config 2023-11-25T05:26:25.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.483+0000 7f43a6e51640 10 monclient: build_initial_monmap 2023-11-25T05:26:25.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.483+0000 7f43a6e51640 10 monclient: monmap: 2023-11-25T05:26:25.485 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:25.486 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:25.486 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:25.484649+0000 2023-11-25T05:26:25.486 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:25.484649+0000 2023-11-25T05:26:25.486 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:25.486 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:25.487 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:25.487 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:25.487 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:25.487 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:25.487 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:25.487 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:25.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.484+0000 7f43a6e51640 1 Processor -- start 2023-11-25T05:26:25.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.484+0000 7f43a6e51640 1 -- start start 2023-11-25T05:26:25.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.484+0000 7f43a6e51640 10 monclient: init 2023-11-25T05:26:25.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.485+0000 7f43a6e51640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:25.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.485+0000 7f43a6e51640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2023-11-25T05:26:25.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.485+0000 7f43a6e51640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f43a00b1bd0 0x7f43a013bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:25.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.485+0000 7f43a6e51640 10 monclient(hunting): picked mon.noname-a con 0x7f43a00b1bd0 addr v2:172.21.15.7:3300/0 2023-11-25T05:26:25.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.485+0000 7f43a6e51640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43a0072770 addr v1:172.21.15.90:6789/0 2023-11-25T05:26:25.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.485+0000 7f43a6e51640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f43a0141f30 0x7f43a01423c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:25.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.485+0000 7f43a6e51640 10 monclient(hunting): picked mon.noname-c con 0x7f43a0141f30 addr v2:172.21.15.90:3300/0 2023-11-25T05:26:25.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.485+0000 7f43a6e51640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a0070fa0 con 0x7f43a0072770 2023-11-25T05:26:25.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.485+0000 7f43a6e51640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:25.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.485+0000 7f43a6e51640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43a0071e00 con 0x7f43a00b1bd0 2023-11-25T05:26:25.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.485+0000 7f43a6e51640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:25.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.485+0000 7f43a6e51640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43a0072070 con 0x7f43a0141f30 2023-11-25T05:26:25.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.485+0000 7f43a6e51640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:25.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.485+0000 7f43a6e51640 10 monclient(hunting): authenticate will time out at 2711.728273s 2023-11-25T05:26:25.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.486+0000 7f43a4bed640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f43a00b1bd0 0x7f43a013bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:25.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.486+0000 7f43a53ee640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f43a0141f30 0x7f43a01423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:25.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.486+0000 7f439ffff640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f43a0072770 0x7f43a013c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:40622/0 (socket says 172.21.15.7:40622) 2023-11-25T05:26:25.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.486+0000 7f439ffff640 1 -- 172.21.15.7:0/3680780752 learned_addr learned my addr 172.21.15.7:0/3680780752 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:26:25.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.486+0000 7f43a4bed640 10 monclient(hunting): get_auth_request con 0x7f43a00b1bd0 auth_method 0 2023-11-25T05:26:25.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.486+0000 7f43a4bed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:25.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.486+0000 7f43a4bed640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:25.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.486+0000 7f43a4bed640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:25.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.486+0000 7f43a53ee640 10 monclient(hunting): get_auth_request con 0x7f43a0141f30 auth_method 0 2023-11-25T05:26:25.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.486+0000 7f43a53ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:25.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.486+0000 7f43a53ee640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:25.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.486+0000 7f43a53ee640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:25.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.487+0000 7f43a4bed640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:25.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.487+0000 7f43a4bed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:25.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.487+0000 7f43a4bed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:25.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.487+0000 7f43a53ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:25.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.487+0000 7f43a53ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:25.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.487+0000 7f43a53ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:25.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.488+0000 7f43a4bed640 10 monclient(hunting): handle_auth_done global_id 15966 payload 274 2023-11-25T05:26:25.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.488+0000 7f43a4bed640 1 -- 172.21.15.7:0/3680780752 >> v2:172.21.15.90:3300/0 conn(0x7f43a0141f30 msgr2=0x7f43a01423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:25.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.488+0000 7f43a4bed640 1 --2- 172.21.15.7:0/3680780752 >> v2:172.21.15.90:3300/0 conn(0x7f43a0141f30 0x7f43a01423c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.488+0000 7f43a4bed640 1 -- 172.21.15.7:0/3680780752 >> v1:172.21.15.90:6789/0 conn(0x7f43a0072770 legacy=0x7f43a013c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:25.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.488+0000 7f43a4bed640 10 monclient: _finish_hunting 0 2023-11-25T05:26:25.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.488+0000 7f43a4bed640 1 monclient: found mon.noname-a 2023-11-25T05:26:25.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.488+0000 7f43a4bed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:25.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.488+0000 7f43a4bed640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:26:25.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.488+0000 7f43a4bed640 1 -- 172.21.15.7:0/3680780752 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a0070300 con 0x7f43a00b1bd0 2023-11-25T05:26:25.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.490+0000 7f43a4bed640 1 --2- 172.21.15.7:0/3680780752 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43a00b1bd0 0x7f43a013bb40 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7f43a00b11a0 tx=0x7f439402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0e84df40322b1b server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:25.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f439f7fe640 1 -- 172.21.15.7:0/3680780752 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4394031710 con 0x7f43a00b1bd0 2023-11-25T05:26:25.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f439f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:25.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f439f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:26:25.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f439f7fe640 10 monclient: dump: 2023-11-25T05:26:25.496 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:25.496 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:25.496 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.496 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.496 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:25.496 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:25.496 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:25.496 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:25.496 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:25.496 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:25.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f439f7fe640 10 monclient: _finish_auth 0 2023-11-25T05:26:25.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f439f7fe640 10 monclient: _check_auth_tickets 2023-11-25T05:26:25.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f439f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:25.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f439f7fe640 1 -- 172.21.15.7:0/3680780752 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439402f050 con 0x7f43a00b1bd0 2023-11-25T05:26:25.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f439f7fe640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:25.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f439f7fe640 1 -- 172.21.15.7:0/3680780752 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4394031a90 con 0x7f43a00b1bd0 2023-11-25T05:26:25.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f439f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:25.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f439f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:26:25.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f439f7fe640 10 monclient: dump: 2023-11-25T05:26:25.497 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:25.498 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:25.498 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.498 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.498 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:25.498 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:25.498 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:25.498 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:25.498 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:25.498 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:25.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f43a6e51640 5 monclient: authenticate success, global_id 15966 2023-11-25T05:26:25.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f43a6e51640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:25.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.494+0000 7f43a6e51640 4 set_mon_vals no callback set 2023-11-25T05:26:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.495+0000 7f43a6e51640 10 monclient: shutdown 2023-11-25T05:26:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.495+0000 7f43a6e51640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.495+0000 7f43a6e51640 1 -- 172.21.15.7:0/3680780752 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43a00b1bd0 msgr2=0x7f43a013bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.495+0000 7f43a6e51640 1 --2- 172.21.15.7:0/3680780752 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43a00b1bd0 0x7f43a013bb40 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7f43a00b11a0 tx=0x7f439402b7b0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.495+0000 7f43a6e51640 1 -- 172.21.15.7:0/3680780752 shutdown_connections 2023-11-25T05:26:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.495+0000 7f43a6e51640 1 --2- 172.21.15.7:0/3680780752 >> v2:172.21.15.90:3300/0 conn(0x7f43a0141f30 0x7f43a01423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.495+0000 7f43a6e51640 1 --2- 172.21.15.7:0/3680780752 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43a00b1bd0 0x7f43a013bb40 unknown :-1 s=CLOSED pgs=3156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.495+0000 7f43a6e51640 1 -- 172.21.15.7:0/3680780752 >> 172.21.15.7:0/3680780752 conn(0x7f43a00b4d20 msgr2=0x7f43a00b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.495+0000 7f43a6e51640 1 -- 172.21.15.7:0/3680780752 shutdown_connections 2023-11-25T05:26:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.495+0000 7f43a6e51640 1 -- 172.21.15.7:0/3680780752 wait complete. 2023-11-25T05:26:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.496+0000 7f43a6e51640 10 monclient: build_initial_monmap 2023-11-25T05:26:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.496+0000 7f43a6e51640 10 monclient: monmap: 2023-11-25T05:26:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:25.505 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.496+0000 7f43a6e51640 1 librados: starting msgr at 2023-11-25T05:26:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.496+0000 7f43a6e51640 1 librados: starting objecter 2023-11-25T05:26:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.496+0000 7f43a6e51640 1 Processor -- start 2023-11-25T05:26:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.496+0000 7f43a6e51640 1 -- start start 2023-11-25T05:26:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.496+0000 7f43a6e51640 1 librados: setting wanted keys 2023-11-25T05:26:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.496+0000 7f43a6e51640 1 librados: calling monclient init 2023-11-25T05:26:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.496+0000 7f43a6e51640 10 monclient: init 2023-11-25T05:26:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-25T05:26:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f43a0072770 0x7f43a01d81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 10 monclient(hunting): picked mon.noname-c con 0x7f43a0072770 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:25.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43a00b1bd0 0x7f43a013c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:25.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 10 monclient(hunting): picked mon.noname-a con 0x7f43a00b1bd0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:25.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a0141f30 0x7f43a01d8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:25.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 10 monclient(hunting): picked mon.noname-b con 0x7f43a0141f30 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:25.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:25.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a013f630 con 0x7f43a00b1bd0 2023-11-25T05:26:25.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:25.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a013ccf0 con 0x7f43a0141f30 2023-11-25T05:26:25.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:25.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a01d8d70 con 0x7f43a0072770 2023-11-25T05:26:25.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:25.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a6e51640 10 monclient(hunting): authenticate will time out at 2711.740215s 2023-11-25T05:26:25.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f43a4bed640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f43a0072770 0x7f43a01d81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:25.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.497+0000 7f439ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43a00b1bd0 0x7f43a013c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:25.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a53ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a0141f30 0x7f43a01d8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a4bed640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f43a0072770 0x7f43a01d81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38430/0 (socket says 172.21.15.7:38430) 2023-11-25T05:26:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a4bed640 1 -- 172.21.15.7:0/2098033171 learned_addr learned my addr 172.21.15.7:0/2098033171 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a4bed640 10 monclient(hunting): get_auth_request con 0x7f43a0072770 auth_method 0 2023-11-25T05:26:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a4bed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a4bed640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a4bed640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f439ffff640 10 monclient(hunting): get_auth_request con 0x7f43a00b1bd0 auth_method 0 2023-11-25T05:26:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f439ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f439ffff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f439ffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a53ee640 10 monclient(hunting): get_auth_request con 0x7f43a0141f30 auth_method 0 2023-11-25T05:26:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a53ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a53ee640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a53ee640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f439ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f439ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f439ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a4bed640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a4bed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a4bed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a53ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a53ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.498+0000 7f43a53ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.501+0000 7f43a53ee640 10 monclient(hunting): handle_auth_done global_id 21295 payload 824 2023-11-25T05:26:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.501+0000 7f43a53ee640 1 -- 172.21.15.7:0/2098033171 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f43a0072770 msgr2=0x7f43a01d81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.501+0000 7f43a53ee640 1 --2- 172.21.15.7:0/2098033171 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f43a0072770 0x7f43a01d81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.501+0000 7f43a53ee640 1 -- 172.21.15.7:0/2098033171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43a00b1bd0 msgr2=0x7f43a013c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.501+0000 7f43a53ee640 1 --2- 172.21.15.7:0/2098033171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43a00b1bd0 0x7f43a013c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.501+0000 7f43a53ee640 10 monclient: _finish_hunting 0 2023-11-25T05:26:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.501+0000 7f43a53ee640 1 monclient: found mon.noname-b 2023-11-25T05:26:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.501+0000 7f43a53ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.501+0000 7f43a53ee640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:26:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.501+0000 7f43a53ee640 1 -- 172.21.15.7:0/2098033171 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4394009400 con 0x7f43a0141f30 2023-11-25T05:26:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.503+0000 7f43a53ee640 1 --2- 172.21.15.7:0/2098033171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a0141f30 0x7f43a01d8830 secure :-1 s=READY pgs=5168 cs=0 l=1 rev1=1 crypto rx=0x7f439800eba0 tx=0x7f439800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.503+0000 7f439d7fa640 10 client.?.objecter ms_handle_connect 0x7f43a0141f30 2023-11-25T05:26:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.503+0000 7f439d7fa640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.506+0000 7f439d7fa640 1 -- 172.21.15.7:0/2098033171 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439800cd20 con 0x7f43a0141f30 2023-11-25T05:26:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.506+0000 7f439d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.506+0000 7f439d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f439d7fa640 10 monclient: dump: 2023-11-25T05:26:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:25.518 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f439d7fa640 10 monclient: _finish_auth 0 2023-11-25T05:26:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f439d7fa640 10 monclient: _check_auth_tickets 2023-11-25T05:26:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f439d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f439d7fa640 1 -- 172.21.15.7:0/2098033171 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439800f040 con 0x7f43a0141f30 2023-11-25T05:26:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f439d7fa640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f439d7fa640 1 -- 172.21.15.7:0/2098033171 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43980146c0 con 0x7f43a0141f30 2023-11-25T05:26:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f43a6e51640 5 monclient: authenticate success, global_id 21295 2023-11-25T05:26:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f43a6e51640 10 monclient: _renew_subs 2023-11-25T05:26:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f43a6e51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f43a6e51640 1 -- 172.21.15.7:0/2098033171 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a013f1c0 con 0x7f43a0141f30 2023-11-25T05:26:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f439effd640 4 set_mon_vals no callback set 2023-11-25T05:26:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f439d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f439d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f439d7fa640 10 monclient: dump: 2023-11-25T05:26:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:25.522 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f43a6e51640 10 client.21295.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:25.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f43a6e51640 10 monclient: _renew_subs 2023-11-25T05:26:25.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f43a6e51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f43a6e51640 1 -- 172.21.15.7:0/2098033171 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a013e9e0 con 0x7f43a0141f30 2023-11-25T05:26:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f43a6e51640 1 librados: init done 2023-11-25T05:26:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f43a6e51640 10 client.21295.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f43a6e51640 15 client.21295.objecter _session_command_op_assign -1 1 2023-11-25T05:26:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f43a6e51640 20 client.21295.objecter _get_session osd=-1 returning homeless 2023-11-25T05:26:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f43a6e51640 10 client.21295.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f43a6e51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.507+0000 7f43a6e51640 1 -- 172.21.15.7:0/2098033171 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f43a0142b10 con 0x7f43a0141f30 2023-11-25T05:26:25.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.526+0000 7f439d7fa640 1 -- 172.21.15.7:0/2098033171 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4398014820 con 0x7f43a0141f30 2023-11-25T05:26:25.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.526+0000 7f439d7fa640 1 --2- 172.21.15.7:0/2098033171 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f438406c820 0x7f438406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:25.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.526+0000 7f43a4bed640 1 --2- 172.21.15.7:0/2098033171 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f438406c820 0x7f438406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:25.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.526+0000 7f43a4bed640 10 monclient: get_auth_request con 0x7f438406c820 auth_method 0 2023-11-25T05:26:25.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.527+0000 7f43a4bed640 1 --2- 172.21.15.7:0/2098033171 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f438406c820 0x7f438406ece0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f439402b7f0 tx=0x7f4394008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:25.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.527+0000 7f439d7fa640 10 client.21295.objecter ms_handle_connect 0x7f438406c820 2023-11-25T05:26:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 1 -- 172.21.15.7:0/2098033171 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f4398059940 con 0x7f43a0141f30 2023-11-25T05:26:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 10 client.21295.objecter ms_dispatch 0x7f43a0000b90 osd_map(918..918 src has 501..918) v4 2023-11-25T05:26:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 3 client.21295.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 3 client.21295.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 10 client.21295.objecter checking command 1 2023-11-25T05:26:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 1 --2- 172.21.15.7:0/2098033171 >> [v2:172.21.15.90:6818/3560048676,v1:172.21.15.90:6819/3560048676] conn(0x7f43840740a0 0x7f4384076540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:25.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 20 client.21295.objecter _get_session s=0x7f43840739d0 osd=6 3 2023-11-25T05:26:25.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 20 client.21295.objecter put_session s=0x7f43840739d0 osd=6 3 2023-11-25T05:26:25.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 15 client.21295.objecter _session_command_op_remove -1 1 2023-11-25T05:26:25.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 20 client.21295.objecter _get_session s=0x7f43840739d0 osd=6 3 2023-11-25T05:26:25.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 20 client.21295.objecter get_session s=0x7f43840739d0 osd=6 3 2023-11-25T05:26:25.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 15 client.21295.objecter _session_command_op_assign 6 1 2023-11-25T05:26:25.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 20 client.21295.objecter put_session s=0x7f43840739d0 osd=6 4 2023-11-25T05:26:25.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 10 client.21295.objecter _send_command 1 2023-11-25T05:26:25.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 1 -- 172.21.15.7:0/2098033171 --> [v2:172.21.15.90:6818/3560048676,v1:172.21.15.90:6819/3560048676] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4384076c10 con 0x7f43840740a0 2023-11-25T05:26:25.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.528+0000 7f439d7fa640 20 client.21295.objecter dump_active .. 0 homeless 2023-11-25T05:26:25.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.529+0000 7f439ffff640 1 --2- 172.21.15.7:0/2098033171 >> [v2:172.21.15.90:6818/3560048676,v1:172.21.15.90:6819/3560048676] conn(0x7f43840740a0 0x7f4384076540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:25.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.529+0000 7f439ffff640 10 monclient: get_auth_request con 0x7f43840740a0 auth_method 0 2023-11-25T05:26:25.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.530+0000 7f439d7fa640 1 -- 172.21.15.7:0/2098033171 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=918) v2 ==== 24+0+0 (secure 0 0 0) 0x7f439800ce80 con 0x7f43a0141f30 2023-11-25T05:26:25.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.530+0000 7f439d7fa640 10 monclient: handle_get_version_reply finishing 1 version 918 2023-11-25T05:26:25.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.532+0000 7f439ffff640 1 --2- 172.21.15.7:0/2098033171 >> [v2:172.21.15.90:6818/3560048676,v1:172.21.15.90:6819/3560048676] conn(0x7f43840740a0 0x7f4384076540 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:25.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.532+0000 7f439d7fa640 10 client.21295.objecter ms_handle_connect 0x7f43840740a0 2023-11-25T05:26:25.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.569+0000 7f439d7fa640 1 -- 172.21.15.7:0/2098033171 <== osd.6 v2:172.21.15.90:6818/3560048676 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f4384076c10 con 0x7f43840740a0 2023-11-25T05:26:25.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.569+0000 7f439d7fa640 10 client.21295.objecter ms_dispatch 0x7f43a0000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:26:25.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.569+0000 7f439d7fa640 10 client.21295.objecter _finish_command 1 = system:0 2023-11-25T05:26:25.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.569+0000 7f439d7fa640 20 client.21295.objecter put_session s=0x7f43840739d0 osd=6 4 2023-11-25T05:26:25.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.569+0000 7f439d7fa640 15 client.21295.objecter _session_command_op_remove 6 1 2023-11-25T05:26:25.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.582+0000 7f43a6e51640 10 client.21295.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-11-25T05:26:25.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.582+0000 7f43a6e51640 15 client.21295.objecter _session_command_op_assign -1 2 2023-11-25T05:26:25.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.582+0000 7f43a6e51640 20 client.21295.objecter _get_session s=0x7f43840739d0 osd=6 3 2023-11-25T05:26:25.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.582+0000 7f43a6e51640 20 client.21295.objecter put_session s=0x7f43840739d0 osd=6 3 2023-11-25T05:26:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.582+0000 7f43a6e51640 20 client.21295.objecter _get_session s=0x7f43840739d0 osd=6 3 2023-11-25T05:26:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.582+0000 7f43a6e51640 15 client.21295.objecter _session_command_op_remove -1 2 2023-11-25T05:26:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.582+0000 7f43a6e51640 20 client.21295.objecter get_session s=0x7f43840739d0 osd=6 3 2023-11-25T05:26:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.582+0000 7f43a6e51640 15 client.21295.objecter _session_command_op_assign 6 2 2023-11-25T05:26:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.582+0000 7f43a6e51640 20 client.21295.objecter put_session s=0x7f43840739d0 osd=6 4 2023-11-25T05:26:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.582+0000 7f43a6e51640 10 client.21295.objecter _send_command 2 2023-11-25T05:26:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.582+0000 7f43a6e51640 1 -- 172.21.15.7:0/2098033171 --> [v2:172.21.15.90:6818/3560048676,v1:172.21.15.90:6819/3560048676] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f43a0075af0 con 0x7f43840740a0 2023-11-25T05:26:25.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f439d7fa640 1 -- 172.21.15.7:0/2098033171 <== osd.6 v2:172.21.15.90:6818/3560048676 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f43a0075af0 con 0x7f43840740a0 2023-11-25T05:26:25.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f439d7fa640 10 client.21295.objecter ms_dispatch 0x7f43a0000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:26:25.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f439d7fa640 10 client.21295.objecter _finish_command 2 = system:0 2023-11-25T05:26:25.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f439d7fa640 20 client.21295.objecter put_session s=0x7f43840739d0 osd=6 4 2023-11-25T05:26:25.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f439d7fa640 15 client.21295.objecter _session_command_op_remove 6 2 2023-11-25T05:26:25.597 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-11-25T05:26:25.598 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-25T05:26:25.598 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-11-25T05:26:25.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 10 librados: watch_flush enter 2023-11-25T05:26:25.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 10 librados: watch_flush exit 2023-11-25T05:26:25.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 10 client.21295.objecter close_session for osd.6 2023-11-25T05:26:25.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 1 -- 172.21.15.7:0/2098033171 >> [v2:172.21.15.90:6818/3560048676,v1:172.21.15.90:6819/3560048676] conn(0x7f43840740a0 msgr2=0x7f4384076540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:25.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 1 --2- 172.21.15.7:0/2098033171 >> [v2:172.21.15.90:6818/3560048676,v1:172.21.15.90:6819/3560048676] conn(0x7f43840740a0 0x7f4384076540 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 20 client.21295.objecter put_session s=0x7f43840739d0 osd=6 1 2023-11-25T05:26:25.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 20 client.21295.objecter shutdown clearing up homeless session... 2023-11-25T05:26:25.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 10 client.21295.objecter successfully canceled tick 2023-11-25T05:26:25.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 1 -- 172.21.15.7:0/2098033171 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f438406c820 msgr2=0x7f438406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:25.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 1 --2- 172.21.15.7:0/2098033171 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f438406c820 0x7f438406ece0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f439402b7f0 tx=0x7f4394008030 comp rx=0 tx=0).stop 2023-11-25T05:26:25.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 10 monclient: shutdown 2023-11-25T05:26:25.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:25.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 1 -- 172.21.15.7:0/2098033171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a0141f30 msgr2=0x7f43a01d8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:25.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 1 --2- 172.21.15.7:0/2098033171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a0141f30 0x7f43a01d8830 secure :-1 s=READY pgs=5168 cs=0 l=1 rev1=1 crypto rx=0x7f439800eba0 tx=0x7f439800c490 comp rx=0 tx=0).stop 2023-11-25T05:26:25.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a53ee640 1 -- 172.21.15.7:0/2098033171 reap_dead start 2023-11-25T05:26:25.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 1 -- 172.21.15.7:0/2098033171 shutdown_connections 2023-11-25T05:26:25.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 1 -- 172.21.15.7:0/2098033171 >> 172.21.15.7:0/2098033171 conn(0x7f43a00b4d20 msgr2=0x7f43a0140f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:25.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 1 -- 172.21.15.7:0/2098033171 shutdown_connections 2023-11-25T05:26:25.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 1 -- 172.21.15.7:0/2098033171 wait complete. 2023-11-25T05:26:25.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.595+0000 7f43a6e51640 1 librados: shutdown 2023-11-25T05:26:25.607 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-11-25T05:26:25.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.713+0000 7fcee641f640 10 monclient: get_monmap_and_config 2023-11-25T05:26:25.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.713+0000 7fcee641f640 10 monclient: build_initial_monmap 2023-11-25T05:26:25.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.713+0000 7fcee641f640 10 monclient: monmap: 2023-11-25T05:26:25.716 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:25.716 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:25.716 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:25.714229+0000 2023-11-25T05:26:25.717 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:25.714229+0000 2023-11-25T05:26:25.717 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:25.717 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:25.717 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:25.717 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:25.717 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:25.718 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:25.718 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:25.718 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:25.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 1 Processor -- start 2023-11-25T05:26:25.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 1 -- start start 2023-11-25T05:26:25.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 10 monclient: init 2023-11-25T05:26:25.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:25.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-11-25T05:26:25.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcee0060f20 0x7fcee0071b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:25.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 10 monclient(hunting): picked mon.noname-a con 0x7fcee0060f20 addr v2:172.21.15.7:3300/0 2023-11-25T05:26:25.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcee0061300 0x7fcee0145780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:25.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 10 monclient(hunting): picked mon.noname-c con 0x7fcee0061300 addr v2:172.21.15.90:3300/0 2023-11-25T05:26:25.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcee0141ef0 addr v1:172.21.15.7:6789/0 2023-11-25T05:26:25.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee006efd0 con 0x7fcee0141ef0 2023-11-25T05:26:25.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:25.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee013c240 con 0x7fcee0060f20 2023-11-25T05:26:25.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:25.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee0146030 con 0x7fcee0061300 2023-11-25T05:26:25.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:25.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.714+0000 7fcee641f640 10 monclient(hunting): authenticate will time out at 2711.957532s 2023-11-25T05:26:25.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.715+0000 7fcedffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcee0060f20 0x7fcee0071b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:25.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.716+0000 7fcedffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcee0060f20 0x7fcee0071b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43856/0 (socket says 172.21.15.7:43856) 2023-11-25T05:26:25.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.716+0000 7fcedffff640 1 -- 172.21.15.7:0/3479936331 learned_addr learned my addr 172.21.15.7:0/3479936331 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:25.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.716+0000 7fcedffff640 10 monclient(hunting): get_auth_request con 0x7fcee0060f20 auth_method 0 2023-11-25T05:26:25.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.716+0000 7fcedffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:25.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.716+0000 7fcedffff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:25.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.716+0000 7fcedffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:25.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.716+0000 7fcedffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:25.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.716+0000 7fcedffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:25.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.716+0000 7fcedffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:25.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.717+0000 7fcedf7fe640 1 --2- 172.21.15.7:0/3479936331 >> v2:172.21.15.90:3300/0 conn(0x7fcee0061300 0x7fcee0145780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:25.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.718+0000 7fcedffff640 10 monclient(hunting): handle_auth_done global_id 15972 payload 274 2023-11-25T05:26:25.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.718+0000 7fcedffff640 1 -- 172.21.15.7:0/3479936331 >> v2:172.21.15.90:3300/0 conn(0x7fcee0061300 msgr2=0x7fcee0145780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:25.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.718+0000 7fcedffff640 1 --2- 172.21.15.7:0/3479936331 >> v2:172.21.15.90:3300/0 conn(0x7fcee0061300 0x7fcee0145780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.718+0000 7fcedffff640 1 -- 172.21.15.7:0/3479936331 >> v1:172.21.15.7:6789/0 conn(0x7fcee0141ef0 legacy=0x7fcee0142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:25.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.718+0000 7fcedffff640 10 monclient: _finish_hunting 0 2023-11-25T05:26:25.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.718+0000 7fcedffff640 1 monclient: found mon.noname-a 2023-11-25T05:26:25.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.718+0000 7fcedffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:25.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.718+0000 7fcedffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:26:25.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.718+0000 7fcedffff640 1 -- 172.21.15.7:0/3479936331 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee0072080 con 0x7fcee0060f20 2023-11-25T05:26:25.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.721+0000 7fcedffff640 1 --2- 172.21.15.7:0/3479936331 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcee0060f20 0x7fcee0071b40 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7fcee00adfe0 tx=0x7fced002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41efe2335018c5f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:25.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.725+0000 7fcedeffd640 1 -- 172.21.15.7:0/3479936331 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0031870 con 0x7fcee0060f20 2023-11-25T05:26:25.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.725+0000 7fcedeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:25.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.725+0000 7fcedeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:26:25.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.725+0000 7fcedeffd640 10 monclient: dump: 2023-11-25T05:26:25.728 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:25.728 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:25.728 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.728 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.728 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:25.729 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.725+0000 7fcedeffd640 10 monclient: _finish_auth 0 2023-11-25T05:26:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.725+0000 7fcedeffd640 10 monclient: _check_auth_tickets 2023-11-25T05:26:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.725+0000 7fcedeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.725+0000 7fcee641f640 5 monclient: authenticate success, global_id 15972 2023-11-25T05:26:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.725+0000 7fcee641f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:26:25.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.743+0000 7fcedeffd640 1 -- 172.21.15.7:0/3479936331 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced00319d0 con 0x7fcee0060f20 2023-11-25T05:26:25.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.743+0000 7fcedeffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:25.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.743+0000 7fcedeffd640 1 -- 172.21.15.7:0/3479936331 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0031d60 con 0x7fcee0060f20 2023-11-25T05:26:25.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.743+0000 7fcedeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:25.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.743+0000 7fcedeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:26:25.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.743+0000 7fcedeffd640 10 monclient: dump: 2023-11-25T05:26:25.745 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:25.745 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:25.745 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.745 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.745 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:25.746 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.743+0000 7fcee641f640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:25.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.743+0000 7fcee641f640 4 set_mon_vals no callback set 2023-11-25T05:26:25.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.744+0000 7fcee641f640 10 monclient: shutdown 2023-11-25T05:26:25.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.744+0000 7fcee641f640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:25.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.744+0000 7fcee641f640 1 -- 172.21.15.7:0/3479936331 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcee0060f20 msgr2=0x7fcee0071b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:25.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.744+0000 7fcee641f640 1 --2- 172.21.15.7:0/3479936331 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcee0060f20 0x7fcee0071b40 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7fcee00adfe0 tx=0x7fced002b910 comp rx=0 tx=0).stop 2023-11-25T05:26:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.744+0000 7fcee641f640 1 -- 172.21.15.7:0/3479936331 shutdown_connections 2023-11-25T05:26:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.744+0000 7fcee641f640 1 --2- 172.21.15.7:0/3479936331 >> v2:172.21.15.90:3300/0 conn(0x7fcee0061300 0x7fcee0145780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.744+0000 7fcee641f640 1 --2- 172.21.15.7:0/3479936331 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcee0060f20 0x7fcee0071b40 unknown :-1 s=CLOSED pgs=3158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.744+0000 7fcee641f640 1 -- 172.21.15.7:0/3479936331 >> 172.21.15.7:0/3479936331 conn(0x7fcee00b3e30 msgr2=0x7fcee00b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.744+0000 7fcee641f640 1 -- 172.21.15.7:0/3479936331 shutdown_connections 2023-11-25T05:26:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.744+0000 7fcee641f640 1 -- 172.21.15.7:0/3479936331 wait complete. 2023-11-25T05:26:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.744+0000 7fcee641f640 10 monclient: build_initial_monmap 2023-11-25T05:26:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.744+0000 7fcee641f640 10 monclient: monmap: 2023-11-25T05:26:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:25.752 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.745+0000 7fcee641f640 1 librados: starting msgr at 2023-11-25T05:26:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.745+0000 7fcee641f640 1 librados: starting objecter 2023-11-25T05:26:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.745+0000 7fcee641f640 1 Processor -- start 2023-11-25T05:26:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.745+0000 7fcee641f640 1 -- start start 2023-11-25T05:26:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.745+0000 7fcee641f640 1 librados: setting wanted keys 2023-11-25T05:26:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.745+0000 7fcee641f640 1 librados: calling monclient init 2023-11-25T05:26:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.745+0000 7fcee641f640 10 monclient: init 2023-11-25T05:26:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-25T05:26:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee0060f20 0x7fcee013c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 10 monclient(hunting): picked mon.noname-b con 0x7fcee0060f20 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcee0141ef0 0x7fcee013e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 10 monclient(hunting): picked mon.noname-c con 0x7fcee0141ef0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcee013acb0 0x7fcee013e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 10 monclient(hunting): picked mon.noname-a con 0x7fcee013acb0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee013d5a0 con 0x7fcee013acb0 2023-11-25T05:26:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee013a4d0 con 0x7fcee0060f20 2023-11-25T05:26:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:25.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee013ebe0 con 0x7fcee0141ef0 2023-11-25T05:26:25.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:25.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcee641f640 10 monclient(hunting): authenticate will time out at 2711.989194s 2023-11-25T05:26:25.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcedffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee0060f20 0x7fcee013c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:25.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.746+0000 7fcedf7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcee0141ef0 0x7fcee013e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:25.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcee49bc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcee013acb0 0x7fcee013e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:25.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcedffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee0060f20 0x7fcee013c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:39438/0 (socket says 172.21.15.7:39438) 2023-11-25T05:26:25.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcedffff640 1 -- 172.21.15.7:0/626631974 learned_addr learned my addr 172.21.15.7:0/626631974 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:25.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcedffff640 10 monclient(hunting): get_auth_request con 0x7fcee0060f20 auth_method 0 2023-11-25T05:26:25.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcedffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:25.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcedffff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:25.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcedffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:25.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcedf7fe640 10 monclient(hunting): get_auth_request con 0x7fcee0141ef0 auth_method 0 2023-11-25T05:26:25.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcedf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:25.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcedf7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:25.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcedf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:25.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcee49bc640 10 monclient(hunting): get_auth_request con 0x7fcee013acb0 auth_method 0 2023-11-25T05:26:25.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcee49bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:25.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcee49bc640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:25.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcee49bc640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:25.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcedffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:25.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcedffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:25.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.747+0000 7fcedffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:25.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.748+0000 7fcee49bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:25.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.748+0000 7fcee49bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:25.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.748+0000 7fcee49bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:25.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.748+0000 7fcedf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:25.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.748+0000 7fcedf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:25.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.748+0000 7fcedf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:25.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.750+0000 7fcedffff640 10 monclient(hunting): handle_auth_done global_id 21298 payload 824 2023-11-25T05:26:25.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.750+0000 7fcedffff640 1 -- 172.21.15.7:0/626631974 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcee0141ef0 msgr2=0x7fcee013e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:25.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.750+0000 7fcedffff640 1 --2- 172.21.15.7:0/626631974 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcee0141ef0 0x7fcee013e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.750+0000 7fcedffff640 1 -- 172.21.15.7:0/626631974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcee013acb0 msgr2=0x7fcee013e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:25.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.750+0000 7fcedffff640 1 --2- 172.21.15.7:0/626631974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcee013acb0 0x7fcee013e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:25.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.750+0000 7fcedffff640 10 monclient: _finish_hunting 0 2023-11-25T05:26:25.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.750+0000 7fcedffff640 1 monclient: found mon.noname-b 2023-11-25T05:26:25.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.750+0000 7fcedffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:25.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.750+0000 7fcedffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:26:25.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.750+0000 7fcedffff640 1 -- 172.21.15.7:0/626631974 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced00094d0 con 0x7fcee0060f20 2023-11-25T05:26:25.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.752+0000 7fcedffff640 1 --2- 172.21.15.7:0/626631974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee0060f20 0x7fcee013c600 secure :-1 s=READY pgs=5169 cs=0 l=1 rev1=1 crypto rx=0x7fced0000c00 tx=0x7fced00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:25.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.752+0000 7fcedcff9640 10 client.?.objecter ms_handle_connect 0x7fcee0060f20 2023-11-25T05:26:25.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.752+0000 7fcedcff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:25.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcedcff9640 1 -- 172.21.15.7:0/626631974 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0031840 con 0x7fcee0060f20 2023-11-25T05:26:25.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcedcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:25.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcedcff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:25.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcedcff9640 10 monclient: dump: 2023-11-25T05:26:25.760 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:25.760 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:25.760 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.760 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:25.761 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcedcff9640 10 monclient: _finish_auth 0 2023-11-25T05:26:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcedcff9640 10 monclient: _check_auth_tickets 2023-11-25T05:26:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcedcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcedcff9640 1 -- 172.21.15.7:0/626631974 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced00443f0 con 0x7fcee0060f20 2023-11-25T05:26:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcedcff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcedcff9640 1 -- 172.21.15.7:0/626631974 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced00447a0 con 0x7fcee0060f20 2023-11-25T05:26:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcedcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:25.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcedcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:25.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcedcff9640 10 monclient: dump: 2023-11-25T05:26:25.762 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:25.762 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:25.762 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.762 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:25.763 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcee641f640 5 monclient: authenticate success, global_id 21298 2023-11-25T05:26:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcee641f640 10 monclient: _renew_subs 2023-11-25T05:26:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcee641f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcee641f640 1 -- 172.21.15.7:0/626631974 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee0142c10 con 0x7fcee0060f20 2023-11-25T05:26:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcede7fc640 4 set_mon_vals no callback set 2023-11-25T05:26:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcee641f640 10 client.21298.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcee641f640 10 monclient: _renew_subs 2023-11-25T05:26:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcee641f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcee641f640 1 -- 172.21.15.7:0/626631974 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee013da60 con 0x7fcee0060f20 2023-11-25T05:26:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.756+0000 7fcee641f640 1 librados: init done 2023-11-25T05:26:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.757+0000 7fcee641f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.757+0000 7fcee641f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.757+0000 7fcee641f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.757+0000 7fcee641f640 1 -- 172.21.15.7:0/626631974 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee013a640 con 0x7fcee0060f20 2023-11-25T05:26:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.776+0000 7fcedcff9640 1 -- 172.21.15.7:0/626631974 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fced004d450 con 0x7fcee0060f20 2023-11-25T05:26:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.776+0000 7fcedcff9640 1 --2- 172.21.15.7:0/626631974 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fcecc06c850 0x7fcecc06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.776+0000 7fcedf7fe640 1 --2- 172.21.15.7:0/626631974 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fcecc06c850 0x7fcecc06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:25.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.776+0000 7fcedf7fe640 10 monclient: get_auth_request con 0x7fcecc06c850 auth_method 0 2023-11-25T05:26:25.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.777+0000 7fcedf7fe640 1 --2- 172.21.15.7:0/626631974 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fcecc06c850 0x7fcecc06ed10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcee013cd20 tx=0x7fced4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:25.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.777+0000 7fcedcff9640 10 client.21298.objecter ms_handle_connect 0x7fcecc06c850 2023-11-25T05:26:25.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.778+0000 7fcedcff9640 1 -- 172.21.15.7:0/626631974 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7fced002d040 con 0x7fcee0060f20 2023-11-25T05:26:25.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.778+0000 7fcedcff9640 10 client.21298.objecter ms_dispatch 0x7fcee0000b90 osd_map(918..918 src has 501..918) v4 2023-11-25T05:26:25.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.778+0000 7fcedcff9640 3 client.21298.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:25.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.778+0000 7fcedcff9640 3 client.21298.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:25.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:25.778+0000 7fcedcff9640 20 client.21298.objecter dump_active .. 0 homeless 2023-11-25T05:26:26.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.012+0000 7fcedcff9640 1 -- 172.21.15.7:0/626631974 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fced00771a0 con 0x7fcee0060f20 2023-11-25T05:26:26.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.012+0000 7fcedcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:26.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.012+0000 7fcedcff9640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:26:26.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.191+0000 7fcee641f640 1 -- 172.21.15.7:0/626631974 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fcee0142a90 con 0x7fcecc06c850 2023-11-25T05:26:26.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.192+0000 7fcedcff9640 1 -- 172.21.15.7:0/626631974 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fced400c040 con 0x7fcecc06c850 2023-11-25T05:26:26.193 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 6 to deep-scrub 2023-11-25T05:26:26.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 10 librados: watch_flush enter 2023-11-25T05:26:26.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 10 librados: watch_flush exit 2023-11-25T05:26:26.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 20 client.21298.objecter shutdown clearing up homeless session... 2023-11-25T05:26:26.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 10 client.21298.objecter successfully canceled tick 2023-11-25T05:26:26.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 1 -- 172.21.15.7:0/626631974 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fcecc06c850 msgr2=0x7fcecc06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:26.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 1 --2- 172.21.15.7:0/626631974 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fcecc06c850 0x7fcecc06ed10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcee013cd20 tx=0x7fced4008040 comp rx=0 tx=0).stop 2023-11-25T05:26:26.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 10 monclient: shutdown 2023-11-25T05:26:26.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:26.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 1 -- 172.21.15.7:0/626631974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee0060f20 msgr2=0x7fcee013c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:26.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 1 --2- 172.21.15.7:0/626631974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee0060f20 0x7fcee013c600 secure :-1 s=READY pgs=5169 cs=0 l=1 rev1=1 crypto rx=0x7fced0000c00 tx=0x7fced00046c0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 1 -- 172.21.15.7:0/626631974 shutdown_connections 2023-11-25T05:26:26.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 1 --2- 172.21.15.7:0/626631974 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fcecc06c850 0x7fcecc06ed10 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 1 --2- 172.21.15.7:0/626631974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcee013acb0 0x7fcee013e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 1 --2- 172.21.15.7:0/626631974 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcee0141ef0 0x7fcee013e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 1 --2- 172.21.15.7:0/626631974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee0060f20 0x7fcee013c600 unknown :-1 s=CLOSED pgs=5169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 1 -- 172.21.15.7:0/626631974 >> 172.21.15.7:0/626631974 conn(0x7fcee00b3e30 msgr2=0x7fcee0073c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:26.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 1 -- 172.21.15.7:0/626631974 shutdown_connections 2023-11-25T05:26:26.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 1 -- 172.21.15.7:0/626631974 wait complete. 2023-11-25T05:26:26.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.194+0000 7fcee641f640 1 librados: shutdown 2023-11-25T05:26:26.208 INFO:tasks.ceph:Scrubbing osd.7 2023-11-25T05:26:26.208 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-11-25T05:26:26.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.323+0000 7fdb0a98a640 10 monclient: get_monmap_and_config 2023-11-25T05:26:26.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.323+0000 7fdb0a98a640 10 monclient: build_initial_monmap 2023-11-25T05:26:26.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.323+0000 7fdb0a98a640 10 monclient: monmap: 2023-11-25T05:26:26.325 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:26.325 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:26.325 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:26.324698+0000 2023-11-25T05:26:26.325 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:26.324698+0000 2023-11-25T05:26:26.325 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:26.325 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:26.325 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:26.326 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:26.326 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:26.326 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:26.326 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:26.326 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:26.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.324+0000 7fdb0a98a640 1 Processor -- start 2023-11-25T05:26:26.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.324+0000 7fdb0a98a640 1 -- start start 2023-11-25T05:26:26.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.324+0000 7fdb0a98a640 10 monclient: init 2023-11-25T05:26:26.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.325+0000 7fdb0a98a640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:26.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.325+0000 7fdb0a98a640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-11-25T05:26:26.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.325+0000 7fdb0a98a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdb04145840 0x7fdb04146080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:26.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.325+0000 7fdb0a98a640 10 monclient(hunting): picked mon.noname-a con 0x7fdb04145840 addr v2:172.21.15.7:3300/0 2023-11-25T05:26:26.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.325+0000 7fdb0a98a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb040b20f0 addr v1:172.21.15.7:6789/0 2023-11-25T05:26:26.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.325+0000 7fdb0a98a640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdb040b2520 0x7fdb0414c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:26.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.325+0000 7fdb0a98a640 10 monclient(hunting): picked mon.noname-c con 0x7fdb040b2520 addr v2:172.21.15.90:3300/0 2023-11-25T05:26:26.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.325+0000 7fdb0a98a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb040aa850 con 0x7fdb040b20f0 2023-11-25T05:26:26.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.325+0000 7fdb0a98a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:26.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.325+0000 7fdb0a98a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb041465c0 con 0x7fdb04145840 2023-11-25T05:26:26.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.325+0000 7fdb0a98a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:26.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.325+0000 7fdb0a98a640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb04145c20 con 0x7fdb040b2520 2023-11-25T05:26:26.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.325+0000 7fdb0a98a640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:26.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.325+0000 7fdb0a98a640 10 monclient(hunting): authenticate will time out at 2712.568133s 2023-11-25T05:26:26.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.325+0000 7fdb08f27640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdb040b2520 0x7fdb0414c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:26.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.326+0000 7fdb08f27640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdb040b2520 0x7fdb0414c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:39442/0 (socket says 172.21.15.7:39442) 2023-11-25T05:26:26.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.326+0000 7fdb08f27640 1 -- 172.21.15.7:0/2434053901 learned_addr learned my addr 172.21.15.7:0/2434053901 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:26.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.326+0000 7fdb08f27640 10 monclient(hunting): get_auth_request con 0x7fdb040b2520 auth_method 0 2023-11-25T05:26:26.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.326+0000 7fdb08f27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:26.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.326+0000 7fdb08f27640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:26.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.326+0000 7fdb08f27640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:26.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.326+0000 7fdb08f27640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:26.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.326+0000 7fdb08f27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:26.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.326+0000 7fdb08f27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:26.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.328+0000 7fdb08f27640 10 monclient(hunting): handle_auth_done global_id 21301 payload 274 2023-11-25T05:26:26.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.328+0000 7fdb08f27640 1 -- 172.21.15.7:0/2434053901 >> v2:172.21.15.7:3300/0 conn(0x7fdb04145840 msgr2=0x7fdb04146080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:26.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.328+0000 7fdb08f27640 1 --2- 172.21.15.7:0/2434053901 >> v2:172.21.15.7:3300/0 conn(0x7fdb04145840 0x7fdb04146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.328+0000 7fdb08f27640 1 -- 172.21.15.7:0/2434053901 >> v1:172.21.15.7:6789/0 conn(0x7fdb040b20f0 legacy=0x7fdb0413f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:26.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.328+0000 7fdb08f27640 10 monclient: _finish_hunting 0 2023-11-25T05:26:26.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.328+0000 7fdb08f27640 1 monclient: found mon.noname-c 2023-11-25T05:26:26.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.328+0000 7fdb08f27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:26.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.328+0000 7fdb08f27640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:26:26.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.328+0000 7fdb08f27640 1 -- 172.21.15.7:0/2434053901 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb040a9260 con 0x7fdb040b2520 2023-11-25T05:26:26.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.330+0000 7fdb08f27640 1 --2- 172.21.15.7:0/2434053901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb040b2520 0x7fdb0414c960 secure :-1 s=READY pgs=5170 cs=0 l=1 rev1=1 crypto rx=0x7fdb040aa490 tx=0x7fdafc02be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9eeef5b738d25c1d server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:26.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb02ffd640 1 -- 172.21.15.7:0/2434053901 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc02c9f0 con 0x7fdb040b2520 2023-11-25T05:26:26.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:26.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:26:26.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb02ffd640 10 monclient: dump: 2023-11-25T05:26:26.340 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:26.340 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:26.340 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.340 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.340 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:26.340 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:26.340 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:26.340 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:26.341 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:26.341 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:26.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb02ffd640 10 monclient: _finish_auth 0 2023-11-25T05:26:26.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb02ffd640 10 monclient: _check_auth_tickets 2023-11-25T05:26:26.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:26.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb02ffd640 1 -- 172.21.15.7:0/2434053901 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdafc02cb50 con 0x7fdb040b2520 2023-11-25T05:26:26.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb02ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:26.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb02ffd640 1 -- 172.21.15.7:0/2434053901 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc0343f0 con 0x7fdb040b2520 2023-11-25T05:26:26.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:26.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:26.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb02ffd640 10 monclient: dump: 2023-11-25T05:26:26.341 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:26.341 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:26.341 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.342 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.342 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:26.342 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:26.342 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:26.342 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:26.342 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:26.342 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:26.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb0a98a640 5 monclient: authenticate success, global_id 21301 2023-11-25T05:26:26.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb0a98a640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:26.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb0a98a640 4 set_mon_vals no callback set 2023-11-25T05:26:26.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb0a98a640 10 monclient: shutdown 2023-11-25T05:26:26.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb0a98a640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:26.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb0a98a640 1 -- 172.21.15.7:0/2434053901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb040b2520 msgr2=0x7fdb0414c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:26.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb0a98a640 1 --2- 172.21.15.7:0/2434053901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb040b2520 0x7fdb0414c960 secure :-1 s=READY pgs=5170 cs=0 l=1 rev1=1 crypto rx=0x7fdb040aa490 tx=0x7fdafc02be50 comp rx=0 tx=0).stop 2023-11-25T05:26:26.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb0a98a640 1 -- 172.21.15.7:0/2434053901 shutdown_connections 2023-11-25T05:26:26.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb0a98a640 1 --2- 172.21.15.7:0/2434053901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb040b2520 0x7fdb0414c960 unknown :-1 s=CLOSED pgs=5170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb0a98a640 1 --2- 172.21.15.7:0/2434053901 >> v2:172.21.15.7:3300/0 conn(0x7fdb04145840 0x7fdb04146080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb0a98a640 1 -- 172.21.15.7:0/2434053901 >> 172.21.15.7:0/2434053901 conn(0x7fdb040b41f0 msgr2=0x7fdb040b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:26.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb0a98a640 1 -- 172.21.15.7:0/2434053901 shutdown_connections 2023-11-25T05:26:26.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.334+0000 7fdb0a98a640 1 -- 172.21.15.7:0/2434053901 wait complete. 2023-11-25T05:26:26.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.335+0000 7fdb0a98a640 10 monclient: build_initial_monmap 2023-11-25T05:26:26.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.335+0000 7fdb0a98a640 10 monclient: monmap: 2023-11-25T05:26:26.343 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:26.343 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:26.343 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:26.344 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.335+0000 7fdb0a98a640 1 librados: starting msgr at 2023-11-25T05:26:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.335+0000 7fdb0a98a640 1 librados: starting objecter 2023-11-25T05:26:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.335+0000 7fdb0a98a640 1 Processor -- start 2023-11-25T05:26:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 1 -- start start 2023-11-25T05:26:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 1 librados: setting wanted keys 2023-11-25T05:26:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 1 librados: calling monclient init 2023-11-25T05:26:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 10 monclient: init 2023-11-25T05:26:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-25T05:26:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb040b20f0 0x7fdb04139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 10 monclient(hunting): picked mon.noname-a con 0x7fdb040b20f0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb040b2520 0x7fdb0413bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 10 monclient(hunting): picked mon.noname-b con 0x7fdb040b2520 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdb04145840 0x7fdb0413c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 10 monclient(hunting): picked mon.noname-c con 0x7fdb04145840 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0413c6d0 con 0x7fdb040b20f0 2023-11-25T05:26:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0413b830 con 0x7fdb040b2520 2023-11-25T05:26:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:26.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb041cd390 con 0x7fdb04145840 2023-11-25T05:26:26.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:26.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.336+0000 7fdb0a98a640 10 monclient(hunting): authenticate will time out at 2712.579540s 2023-11-25T05:26:26.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.337+0000 7fdb037fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb040b2520 0x7fdb0413bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:26.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.337+0000 7fdb08f27640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdb04145840 0x7fdb0413c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:26.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.337+0000 7fdb08f27640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdb04145840 0x7fdb0413c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38448/0 (socket says 172.21.15.7:38448) 2023-11-25T05:26:26.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.337+0000 7fdb08f27640 1 -- 172.21.15.7:0/1758172727 learned_addr learned my addr 172.21.15.7:0/1758172727 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:26.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.337+0000 7fdb08f27640 10 monclient(hunting): get_auth_request con 0x7fdb04145840 auth_method 0 2023-11-25T05:26:26.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.337+0000 7fdb08f27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:26.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.337+0000 7fdb08f27640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:26.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.337+0000 7fdb08f27640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:26.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.337+0000 7fdb037fe640 10 monclient(hunting): get_auth_request con 0x7fdb040b2520 auth_method 0 2023-11-25T05:26:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.337+0000 7fdb037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.337+0000 7fdb037fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.337+0000 7fdb037fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.338+0000 7fdb08f27640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.338+0000 7fdb08f27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.338+0000 7fdb08f27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.338+0000 7fdb037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.338+0000 7fdb037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.338+0000 7fdb037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.341+0000 7fdb03fff640 1 --2- 172.21.15.7:0/1758172727 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb040b20f0 0x7fdb04139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.341+0000 7fdb037fe640 10 monclient(hunting): handle_auth_done global_id 21304 payload 824 2023-11-25T05:26:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.341+0000 7fdb037fe640 1 -- 172.21.15.7:0/1758172727 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdb04145840 msgr2=0x7fdb0413c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.341+0000 7fdb037fe640 1 --2- 172.21.15.7:0/1758172727 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdb04145840 0x7fdb0413c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.341+0000 7fdb037fe640 1 -- 172.21.15.7:0/1758172727 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb040b20f0 msgr2=0x7fdb04139fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.341+0000 7fdb037fe640 1 --2- 172.21.15.7:0/1758172727 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb040b20f0 0x7fdb04139fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.341+0000 7fdb037fe640 10 monclient: _finish_hunting 0 2023-11-25T05:26:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.341+0000 7fdb037fe640 1 monclient: found mon.noname-b 2023-11-25T05:26:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.341+0000 7fdb037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.341+0000 7fdb037fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:26:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.341+0000 7fdb037fe640 1 -- 172.21.15.7:0/1758172727 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdafc009b30 con 0x7fdb040b2520 2023-11-25T05:26:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.343+0000 7fdb037fe640 1 --2- 172.21.15.7:0/1758172727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb040b2520 0x7fdb0413bc50 secure :-1 s=READY pgs=5171 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400c410 tx=0x7fdaf400c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.343+0000 7fdb00ff9640 10 client.?.objecter ms_handle_connect 0x7fdb040b2520 2023-11-25T05:26:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.343+0000 7fdb00ff9640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb00ff9640 1 -- 172.21.15.7:0/1758172727 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf40035f0 con 0x7fdb040b2520 2023-11-25T05:26:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb00ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb00ff9640 10 monclient: dump: 2023-11-25T05:26:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:26.351 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb00ff9640 10 monclient: _finish_auth 0 2023-11-25T05:26:26.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb00ff9640 10 monclient: _check_auth_tickets 2023-11-25T05:26:26.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:26.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb00ff9640 1 -- 172.21.15.7:0/1758172727 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf4003770 con 0x7fdb040b2520 2023-11-25T05:26:26.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb00ff9640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:26.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb00ff9640 1 -- 172.21.15.7:0/1758172727 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4003b60 con 0x7fdb040b2520 2023-11-25T05:26:26.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb0a98a640 5 monclient: authenticate success, global_id 21304 2023-11-25T05:26:26.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb0a98a640 10 monclient: _renew_subs 2023-11-25T05:26:26.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb0a98a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:26.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb0a98a640 1 -- 172.21.15.7:0/1758172727 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb041cdf60 con 0x7fdb040b2520 2023-11-25T05:26:26.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:26.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb027fc640 4 set_mon_vals no callback set 2023-11-25T05:26:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb00ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb00ff9640 10 monclient: dump: 2023-11-25T05:26:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:26.353 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb0a98a640 10 client.21304.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb0a98a640 10 monclient: _renew_subs 2023-11-25T05:26:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb0a98a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb0a98a640 1 -- 172.21.15.7:0/1758172727 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0413a4e0 con 0x7fdb040b2520 2023-11-25T05:26:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.347+0000 7fdb0a98a640 1 librados: init done 2023-11-25T05:26:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.348+0000 7fdb0a98a640 10 client.21304.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.348+0000 7fdb0a98a640 15 client.21304.objecter _session_command_op_assign -1 1 2023-11-25T05:26:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.348+0000 7fdb0a98a640 20 client.21304.objecter _get_session osd=-1 returning homeless 2023-11-25T05:26:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.348+0000 7fdb0a98a640 10 client.21304.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.348+0000 7fdb0a98a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.348+0000 7fdb0a98a640 1 -- 172.21.15.7:0/1758172727 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdb040aad80 con 0x7fdb040b2520 2023-11-25T05:26:26.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.367+0000 7fdb00ff9640 1 -- 172.21.15.7:0/1758172727 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdaf4003cc0 con 0x7fdb040b2520 2023-11-25T05:26:26.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.368+0000 7fdb00ff9640 1 --2- 172.21.15.7:0/1758172727 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdaf006c7d0 0x7fdaf006ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:26.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.368+0000 7fdb03fff640 1 --2- 172.21.15.7:0/1758172727 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdaf006c7d0 0x7fdaf006ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:26.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.368+0000 7fdb03fff640 10 monclient: get_auth_request con 0x7fdaf006c7d0 auth_method 0 2023-11-25T05:26:26.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.368+0000 7fdb03fff640 1 --2- 172.21.15.7:0/1758172727 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdaf006c7d0 0x7fdaf006ec90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800a770 tx=0x7fdaf8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:26.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.368+0000 7fdb00ff9640 10 client.21304.objecter ms_handle_connect 0x7fdaf006c7d0 2023-11-25T05:26:26.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 1 -- 172.21.15.7:0/1758172727 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7fdaf40578f0 con 0x7fdb040b2520 2023-11-25T05:26:26.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 10 client.21304.objecter ms_dispatch 0x7fdb04000b90 osd_map(918..918 src has 501..918) v4 2023-11-25T05:26:26.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 3 client.21304.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:26.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 3 client.21304.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:26.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 10 client.21304.objecter checking command 1 2023-11-25T05:26:26.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 1 --2- 172.21.15.7:0/1758172727 >> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] conn(0x7fdaf0073fc0 0x7fdaf0076460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:26.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 20 client.21304.objecter _get_session s=0x7fdaf00738f0 osd=7 3 2023-11-25T05:26:26.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 20 client.21304.objecter put_session s=0x7fdaf00738f0 osd=7 3 2023-11-25T05:26:26.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 15 client.21304.objecter _session_command_op_remove -1 1 2023-11-25T05:26:26.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 20 client.21304.objecter _get_session s=0x7fdaf00738f0 osd=7 3 2023-11-25T05:26:26.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 20 client.21304.objecter get_session s=0x7fdaf00738f0 osd=7 3 2023-11-25T05:26:26.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 15 client.21304.objecter _session_command_op_assign 7 1 2023-11-25T05:26:26.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 20 client.21304.objecter put_session s=0x7fdaf00738f0 osd=7 4 2023-11-25T05:26:26.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 10 client.21304.objecter _send_command 1 2023-11-25T05:26:26.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 1 -- 172.21.15.7:0/1758172727 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdaf0076b30 con 0x7fdaf0073fc0 2023-11-25T05:26:26.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.370+0000 7fdb00ff9640 20 client.21304.objecter dump_active .. 0 homeless 2023-11-25T05:26:26.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.371+0000 7fdb08f27640 1 --2- 172.21.15.7:0/1758172727 >> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] conn(0x7fdaf0073fc0 0x7fdaf0076460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:26.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.371+0000 7fdb08f27640 10 monclient: get_auth_request con 0x7fdaf0073fc0 auth_method 0 2023-11-25T05:26:26.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.374+0000 7fdb08f27640 1 --2- 172.21.15.7:0/1758172727 >> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] conn(0x7fdaf0073fc0 0x7fdaf0076460 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:26.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.374+0000 7fdb00ff9640 10 client.21304.objecter ms_handle_connect 0x7fdaf0073fc0 2023-11-25T05:26:26.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.397+0000 7fdb00ff9640 1 -- 172.21.15.7:0/1758172727 <== osd.7 v2:172.21.15.90:6802/815268397 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fdaf0076b30 con 0x7fdaf0073fc0 2023-11-25T05:26:26.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.397+0000 7fdb00ff9640 10 client.21304.objecter ms_dispatch 0x7fdb04000b90 command_reply(tid 1: 0 ) v1 2023-11-25T05:26:26.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.397+0000 7fdb00ff9640 10 client.21304.objecter _finish_command 1 = system:0 2023-11-25T05:26:26.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.397+0000 7fdb00ff9640 20 client.21304.objecter put_session s=0x7fdaf00738f0 osd=7 4 2023-11-25T05:26:26.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.397+0000 7fdb00ff9640 15 client.21304.objecter _session_command_op_remove 7 1 2023-11-25T05:26:26.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.411+0000 7fdb0a98a640 10 client.21304.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-11-25T05:26:26.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.411+0000 7fdb0a98a640 15 client.21304.objecter _session_command_op_assign -1 2 2023-11-25T05:26:26.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.411+0000 7fdb0a98a640 20 client.21304.objecter _get_session s=0x7fdaf00738f0 osd=7 3 2023-11-25T05:26:26.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.411+0000 7fdb0a98a640 20 client.21304.objecter put_session s=0x7fdaf00738f0 osd=7 3 2023-11-25T05:26:26.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.411+0000 7fdb0a98a640 20 client.21304.objecter _get_session s=0x7fdaf00738f0 osd=7 3 2023-11-25T05:26:26.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.411+0000 7fdb0a98a640 15 client.21304.objecter _session_command_op_remove -1 2 2023-11-25T05:26:26.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.411+0000 7fdb0a98a640 20 client.21304.objecter get_session s=0x7fdaf00738f0 osd=7 3 2023-11-25T05:26:26.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.411+0000 7fdb0a98a640 15 client.21304.objecter _session_command_op_assign 7 2 2023-11-25T05:26:26.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.411+0000 7fdb0a98a640 20 client.21304.objecter put_session s=0x7fdaf00738f0 osd=7 4 2023-11-25T05:26:26.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.411+0000 7fdb0a98a640 10 client.21304.objecter _send_command 2 2023-11-25T05:26:26.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.411+0000 7fdb0a98a640 1 -- 172.21.15.7:0/1758172727 --> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdb040aaaa0 con 0x7fdaf0073fc0 2023-11-25T05:26:26.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.424+0000 7fdb00ff9640 1 -- 172.21.15.7:0/1758172727 <== osd.7 v2:172.21.15.90:6802/815268397 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fdafc041420 con 0x7fdaf0073fc0 2023-11-25T05:26:26.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.424+0000 7fdb00ff9640 10 client.21304.objecter ms_dispatch 0x7fdb04000b90 command_reply(tid 2: 0 ) v1 2023-11-25T05:26:26.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.424+0000 7fdb00ff9640 10 client.21304.objecter _finish_command 2 = system:0 2023-11-25T05:26:26.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.424+0000 7fdb00ff9640 20 client.21304.objecter put_session s=0x7fdaf00738f0 osd=7 4 2023-11-25T05:26:26.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.424+0000 7fdb00ff9640 15 client.21304.objecter _session_command_op_remove 7 2 2023-11-25T05:26:26.426 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-11-25T05:26:26.427 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-25T05:26:26.427 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-11-25T05:26:26.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 10 librados: watch_flush enter 2023-11-25T05:26:26.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 10 librados: watch_flush exit 2023-11-25T05:26:26.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 10 client.21304.objecter close_session for osd.7 2023-11-25T05:26:26.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 1 -- 172.21.15.7:0/1758172727 >> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] conn(0x7fdaf0073fc0 msgr2=0x7fdaf0076460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:26.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 1 --2- 172.21.15.7:0/1758172727 >> [v2:172.21.15.90:6802/815268397,v1:172.21.15.90:6803/815268397] conn(0x7fdaf0073fc0 0x7fdaf0076460 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 20 client.21304.objecter put_session s=0x7fdaf00738f0 osd=7 1 2023-11-25T05:26:26.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 20 client.21304.objecter shutdown clearing up homeless session... 2023-11-25T05:26:26.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 10 client.21304.objecter successfully canceled tick 2023-11-25T05:26:26.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 1 -- 172.21.15.7:0/1758172727 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdaf006c7d0 msgr2=0x7fdaf006ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:26.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 1 --2- 172.21.15.7:0/1758172727 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fdaf006c7d0 0x7fdaf006ec90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800a770 tx=0x7fdaf8008040 comp rx=0 tx=0).stop 2023-11-25T05:26:26.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 10 monclient: shutdown 2023-11-25T05:26:26.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 20 monclient: shutdown canceling and discarding version request 1 2023-11-25T05:26:26.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:26.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 1 -- 172.21.15.7:0/1758172727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb040b2520 msgr2=0x7fdb0413bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:26.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 1 --2- 172.21.15.7:0/1758172727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb040b2520 0x7fdb0413bc50 secure :-1 s=READY pgs=5171 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400c410 tx=0x7fdaf400c810 comp rx=0 tx=0).stop 2023-11-25T05:26:26.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb08f27640 1 -- 172.21.15.7:0/1758172727 reap_dead start 2023-11-25T05:26:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 1 -- 172.21.15.7:0/1758172727 shutdown_connections 2023-11-25T05:26:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 1 -- 172.21.15.7:0/1758172727 >> 172.21.15.7:0/1758172727 conn(0x7fdb040b41f0 msgr2=0x7fdb040b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 1 -- 172.21.15.7:0/1758172727 shutdown_connections 2023-11-25T05:26:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 1 -- 172.21.15.7:0/1758172727 wait complete. 2023-11-25T05:26:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.425+0000 7fdb0a98a640 1 librados: shutdown 2023-11-25T05:26:26.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-11-25T05:26:26.526 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:08.724 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:26.533 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:08.731 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:26.556 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:08.754 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:26.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.555+0000 7fc2eff5a640 10 monclient: get_monmap_and_config 2023-11-25T05:26:26.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.555+0000 7fc2eff5a640 10 monclient: build_initial_monmap 2023-11-25T05:26:26.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.555+0000 7fc2eff5a640 10 monclient: monmap: 2023-11-25T05:26:26.557 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:26.557 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:26.557 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:26.555966+0000 2023-11-25T05:26:26.557 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:26.555966+0000 2023-11-25T05:26:26.557 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:26.557 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:26.557 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:26.557 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:26.557 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:26.557 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:26.557 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:26.557 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:26.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.555+0000 7fc2eff5a640 1 Processor -- start 2023-11-25T05:26:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.555+0000 7fc2eff5a640 1 -- start start 2023-11-25T05:26:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.555+0000 7fc2eff5a640 10 monclient: init 2023-11-25T05:26:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.556+0000 7fc2eff5a640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.556+0000 7fc2eff5a640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-11-25T05:26:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.556+0000 7fc2eff5a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc2e80a8a30 addr v1:172.21.15.7:6789/0 2023-11-25T05:26:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.556+0000 7fc2eff5a640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc2e8142bf0 0x7fc2e813ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.556+0000 7fc2eff5a640 10 monclient(hunting): picked mon.noname-c con 0x7fc2e8142bf0 addr v2:172.21.15.90:3300/0 2023-11-25T05:26:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.557+0000 7fc2eff5a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc2e80a8e10 0x7fc2e813f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.557+0000 7fc2eff5a640 10 monclient(hunting): picked mon.noname-a con 0x7fc2e80a8e10 addr v2:172.21.15.7:3300/0 2023-11-25T05:26:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.557+0000 7fc2eff5a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e80aac10 con 0x7fc2e80a8a30 2023-11-25T05:26:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.557+0000 7fc2eff5a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.557+0000 7fc2eff5a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e80aaa10 con 0x7fc2e80a8e10 2023-11-25T05:26:26.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.557+0000 7fc2eff5a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:26.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.557+0000 7fc2eff5a640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e80aaef0 con 0x7fc2e8142bf0 2023-11-25T05:26:26.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.557+0000 7fc2eff5a640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:26.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.557+0000 7fc2eff5a640 10 monclient(hunting): authenticate will time out at 2712.799875s 2023-11-25T05:26:26.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.559+0000 7fc2ee4f7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc2e80a8e10 0x7fc2e813f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:26.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.560+0000 7fc2ee4f7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc2e80a8e10 0x7fc2e813f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34394/0 (socket says 172.21.15.7:34394) 2023-11-25T05:26:26.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.560+0000 7fc2ee4f7640 1 -- 172.21.15.7:0/3383838904 learned_addr learned my addr 172.21.15.7:0/3383838904 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:26.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.560+0000 7fc2ee4f7640 10 monclient(hunting): get_auth_request con 0x7fc2e80a8e10 auth_method 0 2023-11-25T05:26:26.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.560+0000 7fc2ee4f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:26.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.560+0000 7fc2ee4f7640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:26.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.560+0000 7fc2ee4f7640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:26.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.562+0000 7fc2ee4f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:26.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.562+0000 7fc2ee4f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:26.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.562+0000 7fc2ee4f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.567+0000 7fc2ee4f7640 10 monclient(hunting): handle_auth_done global_id 15978 payload 274 2023-11-25T05:26:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.567+0000 7fc2ee4f7640 1 -- 172.21.15.7:0/3383838904 >> v2:172.21.15.90:3300/0 conn(0x7fc2e8142bf0 msgr2=0x7fc2e813ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.567+0000 7fc2ee4f7640 1 --2- 172.21.15.7:0/3383838904 >> v2:172.21.15.90:3300/0 conn(0x7fc2e8142bf0 0x7fc2e813ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.567+0000 7fc2ee4f7640 1 -- 172.21.15.7:0/3383838904 >> v1:172.21.15.7:6789/0 conn(0x7fc2e80a8a30 legacy=0x7fc2e81424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.567+0000 7fc2ee4f7640 10 monclient: _finish_hunting 0 2023-11-25T05:26:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.567+0000 7fc2ee4f7640 1 monclient: found mon.noname-a 2023-11-25T05:26:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.567+0000 7fc2ee4f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.567+0000 7fc2ee4f7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:26:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.567+0000 7fc2ee4f7640 1 -- 172.21.15.7:0/3383838904 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e80a97f0 con 0x7fc2e80a8e10 2023-11-25T05:26:26.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.578+0000 7fc2ee4f7640 1 --2- 172.21.15.7:0/3383838904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2e80a8e10 0x7fc2e813f230 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7fc2e80b87d0 tx=0x7fc2e002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7538a409de524960 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:26.585 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:08.729 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:26.586 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:08.731 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:26.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eccf4640 1 -- 172.21.15.7:0/3383838904 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e002cb20 con 0x7fc2e80a8e10 2023-11-25T05:26:26.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eccf4640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:26.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eccf4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:26:26.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eccf4640 10 monclient: dump: 2023-11-25T05:26:26.602 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:26.602 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:26.602 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.602 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.602 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:26.602 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:26.602 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:26.603 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eccf4640 10 monclient: _finish_auth 0 2023-11-25T05:26:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eccf4640 10 monclient: _check_auth_tickets 2023-11-25T05:26:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eccf4640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eccf4640 1 -- 172.21.15.7:0/3383838904 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e002cc80 con 0x7fc2e80a8e10 2023-11-25T05:26:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eff5a640 5 monclient: authenticate success, global_id 15978 2023-11-25T05:26:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eff5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:26:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eccf4640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eff5a640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eccf4640 1 -- 172.21.15.7:0/3383838904 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e00343f0 con 0x7fc2e80a8e10 2023-11-25T05:26:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eff5a640 4 set_mon_vals no callback set 2023-11-25T05:26:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eccf4640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eccf4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:26:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.600+0000 7fc2eccf4640 10 monclient: dump: 2023-11-25T05:26:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:26.605 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:26.605 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:26.605 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:26.606 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:08.747 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:26.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.601+0000 7fc2eff5a640 10 monclient: shutdown 2023-11-25T05:26:26.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.601+0000 7fc2eff5a640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:26.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.601+0000 7fc2eff5a640 1 -- 172.21.15.7:0/3383838904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2e80a8e10 msgr2=0x7fc2e813f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:26.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.601+0000 7fc2eff5a640 1 --2- 172.21.15.7:0/3383838904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2e80a8e10 0x7fc2e813f230 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7fc2e80b87d0 tx=0x7fc2e002bf80 comp rx=0 tx=0).stop 2023-11-25T05:26:26.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.601+0000 7fc2eff5a640 1 -- 172.21.15.7:0/3383838904 shutdown_connections 2023-11-25T05:26:26.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.601+0000 7fc2eff5a640 1 --2- 172.21.15.7:0/3383838904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2e80a8e10 0x7fc2e813f230 unknown :-1 s=CLOSED pgs=3160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.601+0000 7fc2eff5a640 1 --2- 172.21.15.7:0/3383838904 >> v2:172.21.15.90:3300/0 conn(0x7fc2e8142bf0 0x7fc2e813ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.601+0000 7fc2eff5a640 1 -- 172.21.15.7:0/3383838904 >> 172.21.15.7:0/3383838904 conn(0x7fc2e8061a30 msgr2=0x7fc2e80b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:26.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.601+0000 7fc2eff5a640 1 -- 172.21.15.7:0/3383838904 shutdown_connections 2023-11-25T05:26:26.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.601+0000 7fc2eff5a640 1 -- 172.21.15.7:0/3383838904 wait complete. 2023-11-25T05:26:26.610 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:08.801 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:26.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.602+0000 7fc2eff5a640 10 monclient: build_initial_monmap 2023-11-25T05:26:26.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.602+0000 7fc2eff5a640 10 monclient: monmap: 2023-11-25T05:26:26.612 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:26.612 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:26.612 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:26.612 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:26.612 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:26.612 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:26.613 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:26.613 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:26.613 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:26.613 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:26.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.602+0000 7fc2eff5a640 1 librados: starting msgr at 2023-11-25T05:26:26.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.602+0000 7fc2eff5a640 1 librados: starting objecter 2023-11-25T05:26:26.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.602+0000 7fc2eff5a640 1 Processor -- start 2023-11-25T05:26:26.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.602+0000 7fc2eff5a640 1 -- start start 2023-11-25T05:26:26.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.602+0000 7fc2eff5a640 1 librados: setting wanted keys 2023-11-25T05:26:26.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.602+0000 7fc2eff5a640 1 librados: calling monclient init 2023-11-25T05:26:26.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.602+0000 7fc2eff5a640 10 monclient: init 2023-11-25T05:26:26.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:26.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-25T05:26:26.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2e80a8a30 0x7fc2e8143ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:26.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 10 monclient(hunting): picked mon.noname-a con 0x7fc2e80a8a30 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:26.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e80a8e10 0x7fc2e8144a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:26.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 10 monclient(hunting): picked mon.noname-b con 0x7fc2e80a8e10 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:26.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2e8142bf0 0x7fc2e80b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:26.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 10 monclient(hunting): picked mon.noname-c con 0x7fc2e8142bf0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:26.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:26.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e8145320 con 0x7fc2e80a8a30 2023-11-25T05:26:26.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:26.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e81da360 con 0x7fc2e80a8e10 2023-11-25T05:26:26.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:26.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e80b5c90 con 0x7fc2e8142bf0 2023-11-25T05:26:26.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:26.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.603+0000 7fc2eff5a640 10 monclient(hunting): authenticate will time out at 2712.846238s 2023-11-25T05:26:26.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.604+0000 7fc2ee4f7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2e8142bf0 0x7fc2e80b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:26.615 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:08.804 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:26.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.606+0000 7fc2ed4f5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e80a8e10 0x7fc2e8144a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:26.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.606+0000 7fc2edcf6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2e80a8a30 0x7fc2e8143ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:26.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.610+0000 7fc2ed4f5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e80a8e10 0x7fc2e8144a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.7:42120/0 (socket says 172.21.15.7:42120) 2023-11-25T05:26:26.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.610+0000 7fc2ed4f5640 1 -- 172.21.15.7:0/99196879 learned_addr learned my addr 172.21.15.7:0/99196879 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:26.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.610+0000 7fc2ed4f5640 10 monclient(hunting): get_auth_request con 0x7fc2e80a8e10 auth_method 0 2023-11-25T05:26:26.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.610+0000 7fc2ed4f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:26.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.610+0000 7fc2ed4f5640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:26.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.610+0000 7fc2ed4f5640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:26.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.611+0000 7fc2edcf6640 10 monclient(hunting): get_auth_request con 0x7fc2e80a8a30 auth_method 0 2023-11-25T05:26:26.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.611+0000 7fc2edcf6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:26.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.611+0000 7fc2edcf6640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:26.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.611+0000 7fc2edcf6640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:26.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.612+0000 7fc2edcf6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:26.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.612+0000 7fc2edcf6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:26.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.612+0000 7fc2edcf6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:26.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.613+0000 7fc2ed4f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:26.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.613+0000 7fc2ed4f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:26.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.613+0000 7fc2ed4f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:26.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.615+0000 7fc2ee4f7640 10 monclient(hunting): get_auth_request con 0x7fc2e8142bf0 auth_method 0 2023-11-25T05:26:26.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.615+0000 7fc2ee4f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:26.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.615+0000 7fc2ee4f7640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:26.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.615+0000 7fc2ee4f7640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:26.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.616+0000 7fc2ee4f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:26.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.616+0000 7fc2ee4f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:26.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.616+0000 7fc2ee4f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:26.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.616+0000 7fc2edcf6640 10 monclient(hunting): handle_auth_done global_id 15981 payload 824 2023-11-25T05:26:26.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.616+0000 7fc2edcf6640 1 -- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2e8142bf0 msgr2=0x7fc2e80b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:26.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.616+0000 7fc2edcf6640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2e8142bf0 0x7fc2e80b0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.616+0000 7fc2edcf6640 1 -- 172.21.15.7:0/99196879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e80a8e10 msgr2=0x7fc2e8144a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:26.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.616+0000 7fc2edcf6640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e80a8e10 0x7fc2e8144a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.616+0000 7fc2edcf6640 10 monclient: _finish_hunting 0 2023-11-25T05:26:26.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.616+0000 7fc2edcf6640 1 monclient: found mon.noname-a 2023-11-25T05:26:26.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.616+0000 7fc2edcf6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:26.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.616+0000 7fc2edcf6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-11-25T05:26:26.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.616+0000 7fc2edcf6640 1 -- 172.21.15.7:0/99196879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e0009b30 con 0x7fc2e80a8a30 2023-11-25T05:26:26.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.625+0000 7fc2edcf6640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2e80a8a30 0x7fc2e8143ea0 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400c410 tx=0x7fc2e400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:26.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.625+0000 7fc2de7fc640 10 client.?.objecter ms_handle_connect 0x7fc2e80a8a30 2023-11-25T05:26:26.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.625+0000 7fc2de7fc640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:26.630 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:08.828 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:26.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.630+0000 7fc2de7fc640 1 -- 172.21.15.7:0/99196879 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4005230 con 0x7fc2e80a8a30 2023-11-25T05:26:26.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:26.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-25T05:26:26.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2de7fc640 10 monclient: dump: 2023-11-25T05:26:26.634 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:26.634 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:26.634 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.634 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.634 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:26.634 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:26.634 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:26.634 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:26.634 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:26.635 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:26.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2de7fc640 10 monclient: _finish_auth 0 2023-11-25T05:26:26.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2de7fc640 10 monclient: _check_auth_tickets 2023-11-25T05:26:26.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2de7fc640 1 -- 172.21.15.7:0/99196879 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e4005390 con 0x7fc2e80a8a30 2023-11-25T05:26:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2eff5a640 5 monclient: authenticate success, global_id 15981 2023-11-25T05:26:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2eff5a640 10 monclient: _renew_subs 2023-11-25T05:26:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2eff5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:26:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2eff5a640 1 -- 172.21.15.7:0/99196879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e8144fd0 con 0x7fc2e80a8a30 2023-11-25T05:26:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2de7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2de7fc640 1 -- 172.21.15.7:0/99196879 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4004820 con 0x7fc2e80a8a30 2023-11-25T05:26:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-25T05:26:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2de7fc640 10 monclient: dump: 2023-11-25T05:26:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:26.646 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2dffff640 4 set_mon_vals no callback set 2023-11-25T05:26:26.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2eff5a640 10 client.15981.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:26.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2eff5a640 10 monclient: _renew_subs 2023-11-25T05:26:26.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2eff5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:26:26.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2eff5a640 1 -- 172.21.15.7:0/99196879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e81433a0 con 0x7fc2e80a8a30 2023-11-25T05:26:26.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.631+0000 7fc2eff5a640 1 librados: init done 2023-11-25T05:26:26.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.632+0000 7fc2eff5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:26.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.632+0000 7fc2eff5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:26.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.632+0000 7fc2eff5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-11-25T05:26:26.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.632+0000 7fc2eff5a640 1 -- 172.21.15.7:0/99196879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e8067ac0 con 0x7fc2e80a8a30 2023-11-25T05:26:26.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2edcf6640 1 -- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2e80a8a30 msgr2=0x7fc2e8143ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-11-25T05:26:26.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2edcf6640 1 -- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2e80a8a30 msgr2=0x7fc2e8143ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-25T05:26:26.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2edcf6640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2e80a8a30 0x7fc2e8143ea0 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400c410 tx=0x7fc2e400c8d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-25T05:26:26.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2edcf6640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2e80a8a30 0x7fc2e8143ea0 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400c410 tx=0x7fc2e400c8d0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:26.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:26.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 1 -- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2e80a8a30 msgr2=0x7fc2e8143ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-25T05:26:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2e80a8a30 0x7fc2e8143ea0 unknown :-1 s=CLOSED pgs=3161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:26:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2cc005240 0x7fc2cc007630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 10 monclient(hunting): picked mon.a con 0x7fc2cc005240 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2e8142bf0 0x7fc2cc00a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 10 monclient(hunting): picked mon.c con 0x7fc2e8142bf0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e80a8e10 0x7fc2cc00cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 10 monclient(hunting): picked mon.b con 0x7fc2e80a8e10 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 1 -- 172.21.15.7:0/99196879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc00d5b0 con 0x7fc2cc005240 2023-11-25T05:26:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 1 -- 172.21.15.7:0/99196879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc00d720 con 0x7fc2e80a8e10 2023-11-25T05:26:26.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:26.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 1 -- 172.21.15.7:0/99196879 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc00d890 con 0x7fc2e8142bf0 2023-11-25T05:26:26.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.893+0000 7fc2de7fc640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:26.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.895+0000 7fc2ee4f7640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e80a8e10 0x7fc2cc00cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:26.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.895+0000 7fc2ee4f7640 10 monclient(hunting): get_auth_request con 0x7fc2e80a8e10 auth_method 0 2023-11-25T05:26:26.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.895+0000 7fc2ee4f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:26.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.895+0000 7fc2ee4f7640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:26.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.895+0000 7fc2ee4f7640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:26:26.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.897+0000 7fc2ee4f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:26.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.897+0000 7fc2ee4f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:26.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.897+0000 7fc2ee4f7640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-11-25T05:26:26.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.897+0000 7fc2ed4f5640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2e8142bf0 0x7fc2cc00a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:26.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.898+0000 7fc2ed4f5640 10 monclient(hunting): get_auth_request con 0x7fc2e8142bf0 auth_method 0 2023-11-25T05:26:26.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.898+0000 7fc2ed4f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:26.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.898+0000 7fc2ed4f5640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:26.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.898+0000 7fc2ed4f5640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-25T05:26:26.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ed4f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:26.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ed4f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:26.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ed4f5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-11-25T05:26:26.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 10 monclient(hunting): handle_auth_done global_id 15981 payload 293 2023-11-25T05:26:26.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 1 -- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2e8142bf0 msgr2=0x7fc2cc00a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:26.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2e8142bf0 0x7fc2cc00a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 1 -- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2cc005240 msgr2=0x7fc2cc007630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:26.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2cc005240 0x7fc2cc007630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 10 monclient: _finish_hunting 0 2023-11-25T05:26:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 1 monclient: found mon.b 2023-11-25T05:26:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 1 -- 172.21.15.7:0/99196879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fc2cc00db70 con 0x7fc2e80a8e10 2023-11-25T05:26:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 1 -- 172.21.15.7:0/99196879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e80a97f0 con 0x7fc2e80a8e10 2023-11-25T05:26:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 10 monclient: _finish_auth 0 2023-11-25T05:26:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 10 monclient: _check_auth_tickets 2023-11-25T05:26:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.899+0000 7fc2ee4f7640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.906+0000 7fc2de7fc640 10 client.15981.objecter ms_handle_connect 0x7fc2e80a8e10 2023-11-25T05:26:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.906+0000 7fc2de7fc640 10 client.15981.objecter resend_mon_ops 2023-11-25T05:26:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:26.906+0000 7fc2ee4f7640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e80a8e10 0x7fc2cc00cba0 secure :-1 s=READY pgs=5172 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009680 tx=0x7fc2e4007e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:27.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.171+0000 7fc2de7fc640 1 -- 172.21.15.7:0/99196879 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e00344f0 con 0x7fc2e80a8e10 2023-11-25T05:26:27.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.171+0000 7fc2de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.171+0000 7fc2de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.171+0000 7fc2de7fc640 10 monclient: dump: 2023-11-25T05:26:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:27.174 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:27.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.171+0000 7fc2de7fc640 1 -- 172.21.15.7:0/99196879 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e0034650 con 0x7fc2e80a8e10 2023-11-25T05:26:27.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.171+0000 7fc2de7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:27.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.171+0000 7fc2df7fe640 4 set_mon_vals no callback set 2023-11-25T05:26:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.172+0000 7fc2de7fc640 1 -- 172.21.15.7:0/99196879 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc2e00347c0 con 0x7fc2e80a8e10 2023-11-25T05:26:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.172+0000 7fc2de7fc640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc2e80a8a30 0x7fc2cc0770d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.172+0000 7fc2edcf6640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc2e80a8a30 0x7fc2cc0770d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.173+0000 7fc2edcf6640 10 monclient: get_auth_request con 0x7fc2e80a8a30 auth_method 0 2023-11-25T05:26:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.173+0000 7fc2de7fc640 10 client.15981.objecter ms_handle_connect 0x7fc2e80a8a30 2023-11-25T05:26:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.173+0000 7fc2edcf6640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc2e80a8a30 0x7fc2cc0770d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4007d50 tx=0x7fc2e4011000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:27.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.179+0000 7fc2de7fc640 1 -- 172.21.15.7:0/99196879 <== mon.1 v2:172.21.15.90:3300/0 4 ==== osd_map(918..918 src has 687..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7fc2e0079280 con 0x7fc2e80a8e10 2023-11-25T05:26:27.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.179+0000 7fc2de7fc640 10 client.15981.objecter ms_dispatch 0x7fc2e8000b90 osd_map(918..918 src has 687..918) v4 2023-11-25T05:26:27.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.179+0000 7fc2de7fc640 3 client.15981.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:27.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.179+0000 7fc2de7fc640 3 client.15981.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:27.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.179+0000 7fc2de7fc640 20 client.15981.objecter dump_active .. 0 homeless 2023-11-25T05:26:27.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.191+0000 7fc2de7fc640 1 -- 172.21.15.7:0/99196879 <== mon.1 v2:172.21.15.90:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2e00755c0 con 0x7fc2e80a8e10 2023-11-25T05:26:27.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.191+0000 7fc2de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:27.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.191+0000 7fc2de7fc640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:26:27.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.371+0000 7fc2eff5a640 1 -- 172.21.15.7:0/99196879 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2e813f770 con 0x7fc2e80a8a30 2023-11-25T05:26:27.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.372+0000 7fc2de7fc640 1 -- 172.21.15.7:0/99196879 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc2e4004a80 con 0x7fc2e80a8a30 2023-11-25T05:26:27.373 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 7 to deep-scrub 2023-11-25T05:26:27.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 10 librados: watch_flush enter 2023-11-25T05:26:27.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 10 librados: watch_flush exit 2023-11-25T05:26:27.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 20 client.15981.objecter shutdown clearing up homeless session... 2023-11-25T05:26:27.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 10 client.15981.objecter successfully canceled tick 2023-11-25T05:26:27.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 1 -- 172.21.15.7:0/99196879 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc2e80a8a30 msgr2=0x7fc2cc0770d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:27.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc2e80a8a30 0x7fc2cc0770d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4007d50 tx=0x7fc2e4011000 comp rx=0 tx=0).stop 2023-11-25T05:26:27.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 10 monclient: shutdown 2023-11-25T05:26:27.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:27.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 1 -- 172.21.15.7:0/99196879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e80a8e10 msgr2=0x7fc2cc00cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:27.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e80a8e10 0x7fc2cc00cba0 secure :-1 s=READY pgs=5172 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009680 tx=0x7fc2e4007e30 comp rx=0 tx=0).stop 2023-11-25T05:26:27.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 1 -- 172.21.15.7:0/99196879 shutdown_connections 2023-11-25T05:26:27.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7fc2e80a8a30 0x7fc2cc0770d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:27.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e80a8e10 0x7fc2cc00cba0 unknown :-1 s=CLOSED pgs=5172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:27.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2e8142bf0 0x7fc2cc00a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:27.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 1 --2- 172.21.15.7:0/99196879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2cc005240 0x7fc2cc007630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:27.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 1 -- 172.21.15.7:0/99196879 >> 172.21.15.7:0/99196879 conn(0x7fc2e8061a30 msgr2=0x7fc2e81396c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:27.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 1 -- 172.21.15.7:0/99196879 shutdown_connections 2023-11-25T05:26:27.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 1 -- 172.21.15.7:0/99196879 wait complete. 2023-11-25T05:26:27.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.374+0000 7fc2eff5a640 1 librados: shutdown 2023-11-25T05:26:27.387 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-25T05:26:27.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.496+0000 7f849f8c9640 10 monclient: get_monmap_and_config 2023-11-25T05:26:27.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.496+0000 7f849f8c9640 10 monclient: build_initial_monmap 2023-11-25T05:26:27.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.496+0000 7f849f8c9640 10 monclient: monmap: 2023-11-25T05:26:27.499 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:27.499 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:27.499 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:27.497856+0000 2023-11-25T05:26:27.499 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:27.497856+0000 2023-11-25T05:26:27.499 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:27.499 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:27.499 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:27.499 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:27.499 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:27.499 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:27.499 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:27.499 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:27.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.497+0000 7f849f8c9640 1 Processor -- start 2023-11-25T05:26:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.497+0000 7f849f8c9640 1 -- start start 2023-11-25T05:26:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.497+0000 7f849f8c9640 10 monclient: init 2023-11-25T05:26:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.498+0000 7f849f8c9640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.498+0000 7f849f8c9640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-11-25T05:26:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.498+0000 7f849f8c9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f84980a8a30 0x7f84981424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.498+0000 7f849f8c9640 10 monclient(hunting): picked mon.noname-a con 0x7f84980a8a30 addr v2:172.21.15.7:3300/0 2023-11-25T05:26:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.498+0000 7f849f8c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84980a8e10 addr v1:172.21.15.90:6789/0 2023-11-25T05:26:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.498+0000 7f849f8c9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8498142a20 0x7f8498145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.498+0000 7f849f8c9640 10 monclient(hunting): picked mon.noname-b con 0x7f8498142a20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.498+0000 7f849f8c9640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8498142eb0 con 0x7f84980a8e10 2023-11-25T05:26:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.498+0000 7f849f8c9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.498+0000 7f849f8c9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84980aaa10 con 0x7f84980a8a30 2023-11-25T05:26:27.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.498+0000 7f849f8c9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:27.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.498+0000 7f849f8c9640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84980aac10 con 0x7f8498142a20 2023-11-25T05:26:27.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.498+0000 7f849f8c9640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:27.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.498+0000 7f849f8c9640 10 monclient(hunting): authenticate will time out at 2713.741295s 2023-11-25T05:26:27.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.498+0000 7f849de66640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8498142a20 0x7f8498145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:27.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.499+0000 7f849ce64640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f84980a8e10 0x7f84981454a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:51554/0 (socket says 172.21.15.7:51554) 2023-11-25T05:26:27.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.499+0000 7f849ce64640 1 -- 172.21.15.7:0/2909008039 learned_addr learned my addr 172.21.15.7:0/2909008039 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:26:27.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.499+0000 7f849de66640 10 monclient(hunting): get_auth_request con 0x7f8498142a20 auth_method 0 2023-11-25T05:26:27.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.499+0000 7f849de66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:27.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.499+0000 7f849de66640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:27.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.499+0000 7f849de66640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:27.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.499+0000 7f849de66640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:27.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.499+0000 7f849de66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:27.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.499+0000 7f849de66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:27.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.501+0000 7f849de66640 10 monclient(hunting): handle_auth_done global_id 17444 payload 274 2023-11-25T05:26:27.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.501+0000 7f849de66640 1 -- 172.21.15.7:0/2909008039 >> v2:172.21.15.7:3300/0 conn(0x7f84980a8a30 msgr2=0x7f84981424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:26:27.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.501+0000 7f849de66640 1 --2- 172.21.15.7:0/2909008039 >> v2:172.21.15.7:3300/0 conn(0x7f84980a8a30 0x7f84981424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:27.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.501+0000 7f849de66640 1 -- 172.21.15.7:0/2909008039 >> v1:172.21.15.90:6789/0 conn(0x7f84980a8e10 legacy=0x7f84981454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:27.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.501+0000 7f849de66640 10 monclient: _finish_hunting 0 2023-11-25T05:26:27.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.501+0000 7f849de66640 1 monclient: found mon.noname-b 2023-11-25T05:26:27.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.501+0000 7f849de66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:27.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.501+0000 7f849de66640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:26:27.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.501+0000 7f849de66640 1 -- 172.21.15.7:0/2909008039 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84980aaef0 con 0x7f8498142a20 2023-11-25T05:26:27.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.504+0000 7f849de66640 1 --2- 172.21.15.7:0/2909008039 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8498142a20 0x7f8498145bb0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7f8490009d90 tx=0x7f849002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=278830ef9e695f94 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:27.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f848ffff640 1 -- 172.21.15.7:0/2909008039 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849002cce0 con 0x7f8498142a20 2023-11-25T05:26:27.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f848ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:27.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f848ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:27.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f848ffff640 10 monclient: dump: 2023-11-25T05:26:27.510 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:27.510 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:27.510 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:27.510 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:27.511 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:27.511 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:27.511 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:27.511 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:27.511 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:27.511 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:27.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f848ffff640 10 monclient: _finish_auth 0 2023-11-25T05:26:27.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f848ffff640 10 monclient: _check_auth_tickets 2023-11-25T05:26:27.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f848ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:27.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f848ffff640 1 -- 172.21.15.7:0/2909008039 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849002ce40 con 0x7f8498142a20 2023-11-25T05:26:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f848ffff640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f848ffff640 1 -- 172.21.15.7:0/2909008039 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490034660 con 0x7f8498142a20 2023-11-25T05:26:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f849f8c9640 5 monclient: authenticate success, global_id 17444 2023-11-25T05:26:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f849f8c9640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f849f8c9640 4 set_mon_vals no callback set 2023-11-25T05:26:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f848ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f848ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:26:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f848ffff640 10 monclient: dump: 2023-11-25T05:26:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:27.513 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f849f8c9640 10 monclient: shutdown 2023-11-25T05:26:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f849f8c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f849f8c9640 1 -- 172.21.15.7:0/2909008039 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8498142a20 msgr2=0x7f8498145bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f849f8c9640 1 --2- 172.21.15.7:0/2909008039 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8498142a20 0x7f8498145bb0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7f8490009d90 tx=0x7f849002bf30 comp rx=0 tx=0).stop 2023-11-25T05:26:27.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f849f8c9640 1 -- 172.21.15.7:0/2909008039 shutdown_connections 2023-11-25T05:26:27.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f849f8c9640 1 --2- 172.21.15.7:0/2909008039 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8498142a20 0x7f8498145bb0 unknown :-1 s=CLOSED pgs=3073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:27.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f849f8c9640 1 --2- 172.21.15.7:0/2909008039 >> v2:172.21.15.7:3300/0 conn(0x7f84980a8a30 0x7f84981424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:27.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f849f8c9640 1 -- 172.21.15.7:0/2909008039 >> 172.21.15.7:0/2909008039 conn(0x7f8498061a30 msgr2=0x7f84980b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:27.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.508+0000 7f849f8c9640 1 -- 172.21.15.7:0/2909008039 shutdown_connections 2023-11-25T05:26:27.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.509+0000 7f849f8c9640 1 -- 172.21.15.7:0/2909008039 wait complete. 2023-11-25T05:26:27.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.509+0000 7f849f8c9640 10 monclient: build_initial_monmap 2023-11-25T05:26:27.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.509+0000 7f849f8c9640 10 monclient: monmap: 2023-11-25T05:26:27.517 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:27.517 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:27.517 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:27.517 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:27.517 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:27.517 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:27.517 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:27.517 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:27.517 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:27.518 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 1 librados: starting msgr at 2023-11-25T05:26:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 1 librados: starting objecter 2023-11-25T05:26:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 1 Processor -- start 2023-11-25T05:26:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 1 -- start start 2023-11-25T05:26:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 1 librados: setting wanted keys 2023-11-25T05:26:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 1 librados: calling monclient init 2023-11-25T05:26:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 10 monclient: init 2023-11-25T05:26:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-25T05:26:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84980a8a30 0x7f84981da550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 10 monclient(hunting): picked mon.noname-b con 0x7f84980a8a30 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84980a8e10 0x7f84980aee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:27.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 10 monclient(hunting): picked mon.noname-c con 0x7f84980a8e10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:27.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8498142a20 0x7f84980af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 10 monclient(hunting): picked mon.noname-a con 0x7f8498142a20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84980ae970 con 0x7f8498142a20 2023-11-25T05:26:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84980b1470 con 0x7f84980a8a30 2023-11-25T05:26:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84980b1630 con 0x7f84980a8e10 2023-11-25T05:26:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.510+0000 7f849f8c9640 10 monclient(hunting): authenticate will time out at 2713.753816s 2023-11-25T05:26:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.512+0000 7f849de66640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8498142a20 0x7f84980af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.512+0000 7f849ce64640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84980a8e10 0x7f84980aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.512+0000 7f849d665640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84980a8a30 0x7f84981da550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.512+0000 7f849de66640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8498142a20 0x7f84980af3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34442/0 (socket says 172.21.15.7:34442) 2023-11-25T05:26:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.512+0000 7f849de66640 1 -- 172.21.15.7:0/2143322425 learned_addr learned my addr 172.21.15.7:0/2143322425 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.512+0000 7f849de66640 10 monclient(hunting): get_auth_request con 0x7f8498142a20 auth_method 0 2023-11-25T05:26:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.512+0000 7f849de66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.512+0000 7f849de66640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.512+0000 7f849de66640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.513+0000 7f849d665640 10 monclient(hunting): get_auth_request con 0x7f84980a8a30 auth_method 0 2023-11-25T05:26:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.513+0000 7f849d665640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.513+0000 7f849d665640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.513+0000 7f849d665640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.513+0000 7f849ce64640 10 monclient(hunting): get_auth_request con 0x7f84980a8e10 auth_method 0 2023-11-25T05:26:27.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.513+0000 7f849ce64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:27.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.513+0000 7f849ce64640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:27.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.513+0000 7f849ce64640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:27.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.514+0000 7f849de66640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:27.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.514+0000 7f849de66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:27.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.514+0000 7f849de66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:27.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.515+0000 7f849ce64640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:27.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.515+0000 7f849ce64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:27.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.515+0000 7f849ce64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:27.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.517+0000 7f849d665640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:27.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.517+0000 7f849d665640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:27.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.517+0000 7f849d665640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:27.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.521+0000 7f849ce64640 10 monclient(hunting): handle_auth_done global_id 17447 payload 824 2023-11-25T05:26:27.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.521+0000 7f849ce64640 1 -- 172.21.15.7:0/2143322425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84980a8a30 msgr2=0x7f84981da550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:27.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.521+0000 7f849ce64640 1 --2- 172.21.15.7:0/2143322425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84980a8a30 0x7f84981da550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:27.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.521+0000 7f849ce64640 1 -- 172.21.15.7:0/2143322425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8498142a20 msgr2=0x7f84980af3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:27.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.521+0000 7f849ce64640 1 --2- 172.21.15.7:0/2143322425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8498142a20 0x7f84980af3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:27.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.521+0000 7f849ce64640 10 monclient: _finish_hunting 0 2023-11-25T05:26:27.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.521+0000 7f849ce64640 1 monclient: found mon.noname-c 2023-11-25T05:26:27.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.521+0000 7f849ce64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:27.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.521+0000 7f849ce64640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-11-25T05:26:27.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.521+0000 7f849ce64640 1 -- 172.21.15.7:0/2143322425 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8490009a70 con 0x7f84980a8e10 2023-11-25T05:26:27.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.524+0000 7f849ce64640 1 --2- 172.21.15.7:0/2143322425 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84980a8e10 0x7f84980aee90 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f848800c440 tx=0x7f8488001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:27.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.524+0000 7f848dffb640 10 client.?.objecter ms_handle_connect 0x7f84980a8e10 2023-11-25T05:26:27.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.524+0000 7f848dffb640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:27.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f848dffb640 1 -- 172.21.15.7:0/2143322425 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488005ce0 con 0x7f84980a8e10 2023-11-25T05:26:27.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f848dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:27.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f848dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:26:27.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f848dffb640 10 monclient: dump: 2023-11-25T05:26:27.579 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:27.579 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:27.579 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:27.579 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:27.580 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:27.580 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:27.580 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:27.580 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:27.580 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:27.580 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:27.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f848dffb640 10 monclient: _finish_auth 0 2023-11-25T05:26:27.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f848dffb640 10 monclient: _check_auth_tickets 2023-11-25T05:26:27.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f848dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:27.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f849f8c9640 5 monclient: authenticate success, global_id 17447 2023-11-25T05:26:27.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f849f8c9640 10 monclient: _renew_subs 2023-11-25T05:26:27.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f849f8c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:26:27.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f849f8c9640 1 -- 172.21.15.7:0/2143322425 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f849813c380 con 0x7f84980a8e10 2023-11-25T05:26:27.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f849f8c9640 10 client.17447.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:27.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f849f8c9640 10 monclient: _renew_subs 2023-11-25T05:26:27.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f849f8c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:26:27.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f849f8c9640 1 -- 172.21.15.7:0/2143322425 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84980afaf0 con 0x7f84980a8e10 2023-11-25T05:26:27.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.577+0000 7f849f8c9640 1 librados: init done 2023-11-25T05:26:27.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.578+0000 7f849f8c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:27.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.578+0000 7f849f8c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:27.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.578+0000 7f849f8c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-11-25T05:26:27.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.578+0000 7f849f8c9640 1 -- 172.21.15.7:0/2143322425 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8498067ac0 con 0x7f84980a8e10 2023-11-25T05:26:27.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.581+0000 7f848dffb640 1 -- 172.21.15.7:0/2143322425 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8488001120 con 0x7f84980a8e10 2023-11-25T05:26:27.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.581+0000 7f848dffb640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:27.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.581+0000 7f848dffb640 1 -- 172.21.15.7:0/2143322425 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488007270 con 0x7f84980a8e10 2023-11-25T05:26:27.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.581+0000 7f848dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:27.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.581+0000 7f848dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:26:27.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.581+0000 7f848dffb640 10 monclient: dump: 2023-11-25T05:26:27.583 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:27.583 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:27.583 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:27.583 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:27.583 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:27.583 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:27.584 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:27.584 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:27.584 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:27.584 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:27.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.581+0000 7f848f7fe640 4 set_mon_vals no callback set 2023-11-25T05:26:27.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.844+0000 7f848dffb640 1 -- 172.21.15.7:0/2143322425 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f84880073d0 con 0x7f84980a8e10 2023-11-25T05:26:27.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.844+0000 7f848dffb640 1 --2- 172.21.15.7:0/2143322425 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f847c06c800 0x7f847c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:27.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.844+0000 7f849d665640 1 --2- 172.21.15.7:0/2143322425 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f847c06c800 0x7f847c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:27.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.844+0000 7f849d665640 10 monclient: get_auth_request con 0x7f847c06c800 auth_method 0 2023-11-25T05:26:27.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.845+0000 7f849d665640 1 --2- 172.21.15.7:0/2143322425 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f847c06c800 0x7f847c06ecc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8494009990 tx=0x7f8494009110 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:27.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.845+0000 7f848dffb640 10 client.17447.objecter ms_handle_connect 0x7f847c06c800 2023-11-25T05:26:27.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.857+0000 7f848dffb640 1 -- 172.21.15.7:0/2143322425 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(918..918 src has 687..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f8488005e40 con 0x7f84980a8e10 2023-11-25T05:26:27.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.857+0000 7f848dffb640 10 client.17447.objecter ms_dispatch 0x7f8498000b90 osd_map(918..918 src has 687..918) v4 2023-11-25T05:26:27.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.857+0000 7f848dffb640 3 client.17447.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:27.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.857+0000 7f848dffb640 3 client.17447.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:27.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.857+0000 7f848dffb640 20 client.17447.objecter dump_active .. 0 homeless 2023-11-25T05:26:27.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.857+0000 7f848dffb640 1 -- 172.21.15.7:0/2143322425 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8488057ca0 con 0x7f84980a8e10 2023-11-25T05:26:27.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.857+0000 7f848dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:27.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:27.857+0000 7f848dffb640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:26:28.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.033+0000 7f849f8c9640 1 -- 172.21.15.7:0/2143322425 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8498141fc0 con 0x7f847c06c800 2023-11-25T05:26:28.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.039+0000 7f848dffb640 1 -- 172.21.15.7:0/2143322425 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151094 (secure 0 0 0) 0x7f849400e850 con 0x7f847c06c800 2023-11-25T05:26:28.041 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:26:28.042 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-25T05:26:28.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 10 librados: watch_flush enter 2023-11-25T05:26:28.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 10 librados: watch_flush exit 2023-11-25T05:26:28.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 20 client.17447.objecter shutdown clearing up homeless session... 2023-11-25T05:26:28.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 10 client.17447.objecter successfully canceled tick 2023-11-25T05:26:28.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 1 -- 172.21.15.7:0/2143322425 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f847c06c800 msgr2=0x7f847c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:28.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 1 --2- 172.21.15.7:0/2143322425 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f847c06c800 0x7f847c06ecc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8494009990 tx=0x7f8494009110 comp rx=0 tx=0).stop 2023-11-25T05:26:28.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 10 monclient: shutdown 2023-11-25T05:26:28.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:28.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 1 -- 172.21.15.7:0/2143322425 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84980a8e10 msgr2=0x7f84980aee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:28.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 1 --2- 172.21.15.7:0/2143322425 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84980a8e10 0x7f84980aee90 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f848800c440 tx=0x7f8488001e70 comp rx=0 tx=0).stop 2023-11-25T05:26:28.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 1 -- 172.21.15.7:0/2143322425 shutdown_connections 2023-11-25T05:26:28.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 1 --2- 172.21.15.7:0/2143322425 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f847c06c800 0x7f847c06ecc0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:28.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 1 --2- 172.21.15.7:0/2143322425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8498142a20 0x7f84980af3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:28.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 1 --2- 172.21.15.7:0/2143322425 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f84980a8e10 0x7f84980aee90 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:28.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 1 --2- 172.21.15.7:0/2143322425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84980a8a30 0x7f84981da550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:28.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 1 -- 172.21.15.7:0/2143322425 >> 172.21.15.7:0/2143322425 conn(0x7f8498061a30 msgr2=0x7f84980b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:28.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 1 -- 172.21.15.7:0/2143322425 shutdown_connections 2023-11-25T05:26:28.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 1 -- 172.21.15.7:0/2143322425 wait complete. 2023-11-25T05:26:28.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:28.041+0000 7f849f8c9640 1 librados: shutdown 2023-11-25T05:26:28.054 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1585,"stamp":"2023-11-25T05:26:27.715806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":596664,"kb_used_data":107032,"kb_used_omap":12,"kb_used_meta":489459,"kb_avail":754378056,"statfs":{"total":773094113280,"available":772483129344,"internally_reserved":0,"allocated":109600768,"data_stored":67055970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":501206579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":86,"apply_latency_ms":86,"commit_latency_ns":86000000,"apply_latency_ns":86000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.617405"},"pg_stats":[{"pgid":"9.15","version":"0'0","reported_seq":862,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.138807+0000","last_change":"2023-11-25T05:05:46.884752+0000","last_active":"2023-11-25T05:26:16.138807+0000","last_peered":"2023-11-25T05:26:16.138807+0000","last_clean":"2023-11-25T05:26:16.138807+0000","last_became_active":"2023-11-25T05:05:46.875394+0000","last_became_peered":"2023-11-25T05:05:46.875394+0000","last_unstale":"2023-11-25T05:26:16.138807+0000","last_undegraded":"2023-11-25T05:26:16.138807+0000","last_fullsized":"2023-11-25T05:26:16.138807+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":871,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.778823+0000","last_change":"2023-11-25T05:05:46.929874+0000","last_active":"2023-11-25T05:26:16.778823+0000","last_peered":"2023-11-25T05:26:16.778823+0000","last_clean":"2023-11-25T05:26:16.778823+0000","last_became_active":"2023-11-25T05:05:46.923663+0000","last_became_peered":"2023-11-25T05:05:46.923663+0000","last_unstale":"2023-11-25T05:26:16.778823+0000","last_undegraded":"2023-11-25T05:26:16.778823+0000","last_fullsized":"2023-11-25T05:26:16.778823+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":867,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:17.517704+0000","last_change":"2023-11-25T05:05:51.087897+0000","last_active":"2023-11-25T05:26:17.517704+0000","last_peered":"2023-11-25T05:26:17.517704+0000","last_clean":"2023-11-25T05:26:17.517704+0000","last_became_active":"2023-11-25T05:05:50.968719+0000","last_became_peered":"2023-11-25T05:05:50.968719+0000","last_unstale":"2023-11-25T05:26:17.517704+0000","last_undegraded":"2023-11-25T05:26:17.517704+0000","last_fullsized":"2023-11-25T05:26:17.517704+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":860,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:22.444189+0000","last_change":"2023-11-25T05:26:22.444916+0000","last_active":"2023-11-25T05:26:22.444189+0000","last_peered":"2023-11-25T05:26:22.444189+0000","last_clean":"2023-11-25T05:26:22.444189+0000","last_became_active":"2023-11-25T05:06:05.084111+0000","last_became_peered":"2023-11-25T05:06:05.084111+0000","last_unstale":"2023-11-25T05:26:22.444189+0000","last_undegraded":"2023-11-25T05:26:22.444189+0000","last_fullsized":"2023-11-25T05:26:22.444189+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:22.439964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:22.439964+0000","last_clean_scrub_stamp":"2023-11-25T05:26:22.439964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:31:36.581238+0000","scrub_duration":0.053157549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":862,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.110373+0000","last_change":"2023-11-25T05:05:47.892163+0000","last_active":"2023-11-25T05:26:16.110373+0000","last_peered":"2023-11-25T05:26:16.110373+0000","last_clean":"2023-11-25T05:26:16.110373+0000","last_became_active":"2023-11-25T05:05:47.855679+0000","last_became_peered":"2023-11-25T05:05:47.855679+0000","last_unstale":"2023-11-25T05:26:16.110373+0000","last_undegraded":"2023-11-25T05:26:16.110373+0000","last_fullsized":"2023-11-25T05:26:16.110373+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":877,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.135284+0000","last_change":"2023-11-25T05:06:09.646267+0000","last_active":"2023-11-25T05:26:16.135284+0000","last_peered":"2023-11-25T05:26:16.135284+0000","last_clean":"2023-11-25T05:26:16.135284+0000","last_became_active":"2023-11-25T05:06:09.633454+0000","last_became_peered":"2023-11-25T05:06:09.633454+0000","last_unstale":"2023-11-25T05:26:16.135284+0000","last_undegraded":"2023-11-25T05:26:16.135284+0000","last_fullsized":"2023-11-25T05:26:16.135284+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":874,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:23.193489+0000","last_change":"2023-11-25T05:26:23.194252+0000","last_active":"2023-11-25T05:26:23.193489+0000","last_peered":"2023-11-25T05:26:23.193489+0000","last_clean":"2023-11-25T05:26:23.193489+0000","last_became_active":"2023-11-25T05:05:53.099253+0000","last_became_peered":"2023-11-25T05:05:53.099253+0000","last_unstale":"2023-11-25T05:26:23.193489+0000","last_undegraded":"2023-11-25T05:26:23.193489+0000","last_fullsized":"2023-11-25T05:26:23.193489+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:23.192030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:23.192030+0000","last_clean_scrub_stamp":"2023-11-25T05:26:23.192030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:46:00.120705+0000","scrub_duration":0.056747789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":877,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.145292+0000","last_change":"2023-11-25T05:05:43.502596+0000","last_active":"2023-11-25T05:26:16.145292+0000","last_peered":"2023-11-25T05:26:16.145292+0000","last_clean":"2023-11-25T05:26:16.145292+0000","last_became_active":"2023-11-25T05:05:43.496305+0000","last_became_peered":"2023-11-25T05:05:43.496305+0000","last_unstale":"2023-11-25T05:26:16.145292+0000","last_undegraded":"2023-11-25T05:26:16.145292+0000","last_fullsized":"2023-11-25T05:26:16.145292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":872,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.888573+0000","last_change":"2023-11-25T05:05:42.708461+0000","last_active":"2023-11-25T05:26:14.888573+0000","last_peered":"2023-11-25T05:26:14.888573+0000","last_clean":"2023-11-25T05:26:14.888573+0000","last_became_active":"2023-11-25T05:05:42.672388+0000","last_became_peered":"2023-11-25T05:05:42.672388+0000","last_unstale":"2023-11-25T05:26:14.888573+0000","last_undegraded":"2023-11-25T05:26:14.888573+0000","last_fullsized":"2023-11-25T05:26:14.888573+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:34:44.252922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":871,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.780930+0000","last_change":"2023-11-25T05:05:46.966035+0000","last_active":"2023-11-25T05:26:16.780930+0000","last_peered":"2023-11-25T05:26:16.780930+0000","last_clean":"2023-11-25T05:26:16.780930+0000","last_became_active":"2023-11-25T05:05:46.959854+0000","last_became_peered":"2023-11-25T05:05:46.959854+0000","last_unstale":"2023-11-25T05:26:16.780930+0000","last_undegraded":"2023-11-25T05:26:16.780930+0000","last_fullsized":"2023-11-25T05:26:16.780930+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":872,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.898441+0000","last_change":"2023-11-25T05:05:47.759205+0000","last_active":"2023-11-25T05:26:14.898441+0000","last_peered":"2023-11-25T05:26:14.898441+0000","last_clean":"2023-11-25T05:26:14.898441+0000","last_became_active":"2023-11-25T05:05:47.753176+0000","last_became_peered":"2023-11-25T05:05:47.753176+0000","last_unstale":"2023-11-25T05:26:14.898441+0000","last_undegraded":"2023-11-25T05:26:14.898441+0000","last_fullsized":"2023-11-25T05:26:14.898441+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:04:21.487913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":873,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.882638+0000","last_change":"2023-11-25T05:05:45.604133+0000","last_active":"2023-11-25T05:26:14.882638+0000","last_peered":"2023-11-25T05:26:14.882638+0000","last_clean":"2023-11-25T05:26:14.882638+0000","last_became_active":"2023-11-25T05:05:45.597733+0000","last_became_peered":"2023-11-25T05:05:45.597733+0000","last_unstale":"2023-11-25T05:26:14.882638+0000","last_undegraded":"2023-11-25T05:26:14.882638+0000","last_fullsized":"2023-11-25T05:26:14.882638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T08:20:05.883993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.912668+0000","last_change":"2023-11-25T05:05:47.799603+0000","last_active":"2023-11-25T05:26:15.912668+0000","last_peered":"2023-11-25T05:26:15.912668+0000","last_clean":"2023-11-25T05:26:15.912668+0000","last_became_active":"2023-11-25T05:05:47.792727+0000","last_became_peered":"2023-11-25T05:05:47.792727+0000","last_unstale":"2023-11-25T05:26:15.912668+0000","last_undegraded":"2023-11-25T05:26:15.912668+0000","last_fullsized":"2023-11-25T05:26:15.912668+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:50:46.409556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":867,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.141691+0000","last_change":"2023-11-25T05:04:33.582978+0000","last_active":"2023-11-25T05:26:16.141691+0000","last_peered":"2023-11-25T05:26:16.141691+0000","last_clean":"2023-11-25T05:26:16.141691+0000","last_became_active":"2023-11-25T05:04:33.577510+0000","last_became_peered":"2023-11-25T05:04:33.577510+0000","last_unstale":"2023-11-25T05:26:16.141691+0000","last_undegraded":"2023-11-25T05:26:16.141691+0000","last_fullsized":"2023-11-25T05:26:16.141691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":863,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.137153+0000","last_change":"2023-11-25T05:05:58.700776+0000","last_active":"2023-11-25T05:26:16.137153+0000","last_peered":"2023-11-25T05:26:16.137153+0000","last_clean":"2023-11-25T05:26:16.137153+0000","last_became_active":"2023-11-25T05:05:58.695110+0000","last_became_peered":"2023-11-25T05:05:58.695110+0000","last_unstale":"2023-11-25T05:26:16.137153+0000","last_undegraded":"2023-11-25T05:26:16.137153+0000","last_fullsized":"2023-11-25T05:26:16.137153+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":881,"reported_epoch":918,"state":"active+clean+scrubbing+deep","last_fresh":"2023-11-25T05:26:26.912999+0000","last_change":"2023-11-25T05:26:26.900881+0000","last_active":"2023-11-25T05:26:26.912999+0000","last_peered":"2023-11-25T05:26:26.912999+0000","last_clean":"2023-11-25T05:26:26.912999+0000","last_became_active":"2023-11-25T05:04:33.610776+0000","last_became_peered":"2023-11-25T05:04:33.610776+0000","last_unstale":"2023-11-25T05:26:26.912999+0000","last_undegraded":"2023-11-25T05:26:26.912999+0000","last_fullsized":"2023-11-25T05:26:26.912999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":867,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:17.505703+0000","last_change":"2023-11-25T05:06:12.667457+0000","last_active":"2023-11-25T05:26:17.505703+0000","last_peered":"2023-11-25T05:26:17.505703+0000","last_clean":"2023-11-25T05:26:17.505703+0000","last_became_active":"2023-11-25T05:06:12.543810+0000","last_became_peered":"2023-11-25T05:06:12.543810+0000","last_unstale":"2023-11-25T05:26:17.505703+0000","last_undegraded":"2023-11-25T05:26:17.505703+0000","last_fullsized":"2023-11-25T05:26:17.505703+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":868,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.135482+0000","last_change":"2023-11-25T05:04:33.350255+0000","last_active":"2023-11-25T05:26:16.135482+0000","last_peered":"2023-11-25T05:26:16.135482+0000","last_clean":"2023-11-25T05:26:16.135482+0000","last_became_active":"2023-11-25T05:04:33.337158+0000","last_became_peered":"2023-11-25T05:04:33.337158+0000","last_unstale":"2023-11-25T05:26:16.135482+0000","last_undegraded":"2023-11-25T05:26:16.135482+0000","last_fullsized":"2023-11-25T05:26:16.135482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":867,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:17.519430+0000","last_change":"2023-11-25T05:05:51.120038+0000","last_active":"2023-11-25T05:26:17.519430+0000","last_peered":"2023-11-25T05:26:17.519430+0000","last_clean":"2023-11-25T05:26:17.519430+0000","last_became_active":"2023-11-25T05:05:51.097150+0000","last_became_peered":"2023-11-25T05:05:51.097150+0000","last_unstale":"2023-11-25T05:26:17.519430+0000","last_undegraded":"2023-11-25T05:26:17.519430+0000","last_fullsized":"2023-11-25T05:26:17.519430+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":885,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:26.311099+0000","last_change":"2023-11-25T05:26:26.315627+0000","last_active":"2023-11-25T05:26:26.311099+0000","last_peered":"2023-11-25T05:26:26.311099+0000","last_clean":"2023-11-25T05:26:26.311099+0000","last_became_active":"2023-11-25T05:04:33.388849+0000","last_became_peered":"2023-11-25T05:04:33.388849+0000","last_unstale":"2023-11-25T05:26:26.311099+0000","last_undegraded":"2023-11-25T05:26:26.311099+0000","last_fullsized":"2023-11-25T05:26:26.311099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:26.307049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:26.307049+0000","last_clean_scrub_stamp":"2023-11-25T05:26:26.307049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:30:19.433152+0000","scrub_duration":0.21619841400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":872,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.770547+0000","last_change":"2023-11-25T05:05:55.046489+0000","last_active":"2023-11-25T05:26:16.770547+0000","last_peered":"2023-11-25T05:26:16.770547+0000","last_clean":"2023-11-25T05:26:16.770547+0000","last_became_active":"2023-11-25T05:05:55.038711+0000","last_became_peered":"2023-11-25T05:05:55.038711+0000","last_unstale":"2023-11-25T05:26:16.770547+0000","last_undegraded":"2023-11-25T05:26:16.770547+0000","last_fullsized":"2023-11-25T05:26:16.770547+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":868,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.119892+0000","last_change":"2023-11-25T05:04:33.381810+0000","last_active":"2023-11-25T05:26:16.119892+0000","last_peered":"2023-11-25T05:26:16.119892+0000","last_clean":"2023-11-25T05:26:16.119892+0000","last_became_active":"2023-11-25T05:04:33.376375+0000","last_became_peered":"2023-11-25T05:04:33.376375+0000","last_unstale":"2023-11-25T05:26:16.119892+0000","last_undegraded":"2023-11-25T05:26:16.119892+0000","last_fullsized":"2023-11-25T05:26:16.119892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":878,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.132712+0000","last_change":"2023-11-25T05:05:44.569896+0000","last_active":"2023-11-25T05:26:16.132712+0000","last_peered":"2023-11-25T05:26:16.132712+0000","last_clean":"2023-11-25T05:26:16.132712+0000","last_became_active":"2023-11-25T05:05:44.554983+0000","last_became_peered":"2023-11-25T05:05:44.554983+0000","last_unstale":"2023-11-25T05:26:16.132712+0000","last_undegraded":"2023-11-25T05:26:16.132712+0000","last_fullsized":"2023-11-25T05:26:16.132712+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":873,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.904489+0000","last_change":"2023-11-25T05:04:33.238092+0000","last_active":"2023-11-25T05:26:15.904489+0000","last_peered":"2023-11-25T05:26:15.904489+0000","last_clean":"2023-11-25T05:26:15.904489+0000","last_became_active":"2023-11-25T05:04:33.196533+0000","last_became_peered":"2023-11-25T05:04:33.196533+0000","last_unstale":"2023-11-25T05:26:15.904489+0000","last_undegraded":"2023-11-25T05:26:15.904489+0000","last_fullsized":"2023-11-25T05:26:15.904489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:06:22.475692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":872,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.775098+0000","last_change":"2023-11-25T05:05:45.774913+0000","last_active":"2023-11-25T05:26:16.775098+0000","last_peered":"2023-11-25T05:26:16.775098+0000","last_clean":"2023-11-25T05:26:16.775098+0000","last_became_active":"2023-11-25T05:05:45.759824+0000","last_became_peered":"2023-11-25T05:05:45.759824+0000","last_unstale":"2023-11-25T05:26:16.775098+0000","last_undegraded":"2023-11-25T05:26:16.775098+0000","last_fullsized":"2023-11-25T05:26:16.775098+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":868,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.117740+0000","last_change":"2023-11-25T05:04:33.294027+0000","last_active":"2023-11-25T05:26:16.117740+0000","last_peered":"2023-11-25T05:26:16.117740+0000","last_clean":"2023-11-25T05:26:16.117740+0000","last_became_active":"2023-11-25T05:04:33.242068+0000","last_became_peered":"2023-11-25T05:04:33.242068+0000","last_unstale":"2023-11-25T05:26:16.117740+0000","last_undegraded":"2023-11-25T05:26:16.117740+0000","last_fullsized":"2023-11-25T05:26:16.117740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:27.111527+0000","last_change":"2023-11-25T05:26:27.112227+0000","last_active":"2023-11-25T05:26:27.111527+0000","last_peered":"2023-11-25T05:26:27.111527+0000","last_clean":"2023-11-25T05:26:27.111527+0000","last_became_active":"2023-11-25T05:05:45.057546+0000","last_became_peered":"2023-11-25T05:05:45.057546+0000","last_unstale":"2023-11-25T05:26:27.111527+0000","last_undegraded":"2023-11-25T05:26:27.111527+0000","last_fullsized":"2023-11-25T05:26:27.111527+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:27.110149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:27.110149+0000","last_clean_scrub_stamp":"2023-11-25T05:26:27.110149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T17:19:27.768590+0000","scrub_duration":0.035816397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":868,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.127164+0000","last_change":"2023-11-25T05:04:33.784328+0000","last_active":"2023-11-25T05:26:16.127164+0000","last_peered":"2023-11-25T05:26:16.127164+0000","last_clean":"2023-11-25T05:26:16.127164+0000","last_became_active":"2023-11-25T05:04:33.778042+0000","last_became_peered":"2023-11-25T05:04:33.778042+0000","last_unstale":"2023-11-25T05:26:16.127164+0000","last_undegraded":"2023-11-25T05:26:16.127164+0000","last_fullsized":"2023-11-25T05:26:16.127164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:04:31.491894+0000","last_clean_scrub_stamp":"2023-11-25T05:04:31.491894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"918'78","reported_seq":987,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:22.187717+0000","last_change":"2023-11-25T05:26:22.189192+0000","last_active":"2023-11-25T05:26:22.187717+0000","last_peered":"2023-11-25T05:26:22.187717+0000","last_clean":"2023-11-25T05:26:22.187717+0000","last_became_active":"2023-11-25T05:04:27.388156+0000","last_became_peered":"2023-11-25T05:04:27.388156+0000","last_unstale":"2023-11-25T05:26:22.187717+0000","last_undegraded":"2023-11-25T05:26:22.187717+0000","last_fullsized":"2023-11-25T05:26:22.187717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'78","last_scrub_stamp":"2023-11-25T05:26:22.186221+0000","last_deep_scrub":"918'78","last_deep_scrub_stamp":"2023-11-25T05:26:22.186221+0000","last_clean_scrub_stamp":"2023-11-25T05:26:22.186221+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:49:24.005907+0000","scrub_duration":0.15619770899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":873,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.891303+0000","last_change":"2023-11-25T05:05:47.996637+0000","last_active":"2023-11-25T05:26:14.891303+0000","last_peered":"2023-11-25T05:26:14.891303+0000","last_clean":"2023-11-25T05:26:14.891303+0000","last_became_active":"2023-11-25T05:05:47.973359+0000","last_became_peered":"2023-11-25T05:05:47.973359+0000","last_unstale":"2023-11-25T05:26:14.891303+0000","last_undegraded":"2023-11-25T05:26:14.891303+0000","last_fullsized":"2023-11-25T05:26:14.891303+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:05:25.523743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":862,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:21.466002+0000","last_change":"2023-11-25T05:26:21.466727+0000","last_active":"2023-11-25T05:26:21.466002+0000","last_peered":"2023-11-25T05:26:21.466002+0000","last_clean":"2023-11-25T05:26:21.466002+0000","last_became_active":"2023-11-25T05:05:54.045707+0000","last_became_peered":"2023-11-25T05:05:54.045707+0000","last_unstale":"2023-11-25T05:26:21.466002+0000","last_undegraded":"2023-11-25T05:26:21.466002+0000","last_fullsized":"2023-11-25T05:26:21.466002+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:21.462123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:21.462123+0000","last_clean_scrub_stamp":"2023-11-25T05:26:21.462123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:08:12.548363+0000","scrub_duration":0.061865305000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":873,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.798627+0000","last_change":"2023-11-25T05:05:42.506167+0000","last_active":"2023-11-25T05:26:16.798627+0000","last_peered":"2023-11-25T05:26:16.798627+0000","last_clean":"2023-11-25T05:26:16.798627+0000","last_became_active":"2023-11-25T05:05:42.487757+0000","last_became_peered":"2023-11-25T05:05:42.487757+0000","last_unstale":"2023-11-25T05:26:16.798627+0000","last_undegraded":"2023-11-25T05:26:16.798627+0000","last_fullsized":"2023-11-25T05:26:16.798627+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"38'7","reported_seq":881,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.902404+0000","last_change":"2023-11-25T05:05:45.233028+0000","last_active":"2023-11-25T05:26:15.902404+0000","last_peered":"2023-11-25T05:26:15.902404+0000","last_clean":"2023-11-25T05:26:15.902404+0000","last_became_active":"2023-11-25T05:05:45.212870+0000","last_became_peered":"2023-11-25T05:05:45.212870+0000","last_unstale":"2023-11-25T05:26:15.902404+0000","last_undegraded":"2023-11-25T05:26:15.902404+0000","last_fullsized":"2023-11-25T05:26:15.902404+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:44:02.468772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:24.156130+0000","last_change":"2023-11-25T05:26:24.156858+0000","last_active":"2023-11-25T05:26:24.156130+0000","last_peered":"2023-11-25T05:26:24.156130+0000","last_clean":"2023-11-25T05:26:24.156130+0000","last_became_active":"2023-11-25T05:05:49.941593+0000","last_became_peered":"2023-11-25T05:05:49.941593+0000","last_unstale":"2023-11-25T05:26:24.156130+0000","last_undegraded":"2023-11-25T05:26:24.156130+0000","last_fullsized":"2023-11-25T05:26:24.156130+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:24.154721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:24.154721+0000","last_clean_scrub_stamp":"2023-11-25T05:26:24.154721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:11:07.189026+0000","scrub_duration":0.037000584000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":867,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:17.521154+0000","last_change":"2023-11-25T05:06:26.505899+0000","last_active":"2023-11-25T05:26:17.521154+0000","last_peered":"2023-11-25T05:26:17.521154+0000","last_clean":"2023-11-25T05:26:17.521154+0000","last_became_active":"2023-11-25T05:06:26.500023+0000","last_became_peered":"2023-11-25T05:06:26.500023+0000","last_unstale":"2023-11-25T05:26:17.521154+0000","last_undegraded":"2023-11-25T05:26:17.521154+0000","last_fullsized":"2023-11-25T05:26:17.521154+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-11-25T05:26:15.910660+0000","last_change":"2023-11-25T05:05:48.099018+0000","last_active":"2023-11-25T05:26:15.910660+0000","last_peered":"2023-11-25T05:26:15.910660+0000","last_clean":"2023-11-25T05:26:15.910660+0000","last_became_active":"2023-11-25T05:05:48.093083+0000","last_became_peered":"2023-11-25T05:05:48.093083+0000","last_unstale":"2023-11-25T05:26:15.910660+0000","last_undegraded":"2023-11-25T05:26:15.910660+0000","last_fullsized":"2023-11-25T05:26:15.910660+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:36:31.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":871,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.785335+0000","last_change":"2023-11-25T05:05:55.013827+0000","last_active":"2023-11-25T05:26:16.785335+0000","last_peered":"2023-11-25T05:26:16.785335+0000","last_clean":"2023-11-25T05:26:16.785335+0000","last_became_active":"2023-11-25T05:05:55.006070+0000","last_became_peered":"2023-11-25T05:05:55.006070+0000","last_unstale":"2023-11-25T05:26:16.785335+0000","last_undegraded":"2023-11-25T05:26:16.785335+0000","last_fullsized":"2023-11-25T05:26:16.785335+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":871,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.772818+0000","last_change":"2023-11-25T05:05:42.334720+0000","last_active":"2023-11-25T05:26:16.772818+0000","last_peered":"2023-11-25T05:26:16.772818+0000","last_clean":"2023-11-25T05:26:16.772818+0000","last_became_active":"2023-11-25T05:05:42.327418+0000","last_became_peered":"2023-11-25T05:05:42.327418+0000","last_unstale":"2023-11-25T05:26:16.772818+0000","last_undegraded":"2023-11-25T05:26:16.772818+0000","last_fullsized":"2023-11-25T05:26:16.772818+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":871,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:17.447759+0000","last_change":"2023-11-25T05:05:45.341062+0000","last_active":"2023-11-25T05:26:17.447759+0000","last_peered":"2023-11-25T05:26:17.447759+0000","last_clean":"2023-11-25T05:26:17.447759+0000","last_became_active":"2023-11-25T05:05:45.334024+0000","last_became_peered":"2023-11-25T05:05:45.334024+0000","last_unstale":"2023-11-25T05:26:17.447759+0000","last_undegraded":"2023-11-25T05:26:17.447759+0000","last_fullsized":"2023-11-25T05:26:17.447759+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":867,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:17.500159+0000","last_change":"2023-11-25T05:06:27.132506+0000","last_active":"2023-11-25T05:26:17.500159+0000","last_peered":"2023-11-25T05:26:17.500159+0000","last_clean":"2023-11-25T05:26:17.500159+0000","last_became_active":"2023-11-25T05:06:27.122861+0000","last_became_peered":"2023-11-25T05:06:27.122861+0000","last_unstale":"2023-11-25T05:26:17.500159+0000","last_undegraded":"2023-11-25T05:26:17.500159+0000","last_fullsized":"2023-11-25T05:26:17.500159+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":872,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-11-25T05:26:14.875868+0000","last_change":"2023-11-25T05:05:57.191530+0000","last_active":"2023-11-25T05:26:14.875868+0000","last_peered":"2023-11-25T05:26:14.875868+0000","last_clean":"2023-11-25T05:26:14.875868+0000","last_became_active":"2023-11-25T05:05:57.177615+0000","last_became_peered":"2023-11-25T05:05:57.177615+0000","last_unstale":"2023-11-25T05:26:14.875868+0000","last_undegraded":"2023-11-25T05:26:14.875868+0000","last_fullsized":"2023-11-25T05:26:14.875868+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:21:49.038926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":16,"seq":68719477001,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66664,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305176,"statfs":{"total":96636764160,"available":96568500224,"internally_reserved":0,"allocated":13598720,"data_stored":8282535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":17.164000000000001,"5min":25.925000000000001,"15min":25.350999999999999},"min":{"1min":4.9020000000000001,"5min":4.9020000000000001,"15min":4.9020000000000001},"max":{"1min":100.782,"5min":164.57300000000001,"15min":203.28800000000001},"last":9.8789999999999996},{"interface":"front","average":{"1min":15.237,"5min":26.946000000000002,"15min":26.018000000000001},"min":{"1min":5.7000000000000002,"5min":5.6859999999999999,"15min":5.4400000000000004},"max":{"1min":92.989000000000004,"5min":238.51300000000001,"15min":347.452},"last":14.206}]},{"osd":1,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":20.888000000000002,"5min":26.521999999999998,"15min":25.02},"min":{"1min":8.3249999999999993,"5min":7.4660000000000002,"15min":5.6589999999999998},"max":{"1min":86.962999999999994,"5min":287.27199999999999,"15min":287.27199999999999},"last":12.486000000000001},{"interface":"front","average":{"1min":17.350999999999999,"5min":27.222000000000001,"15min":23.896000000000001},"min":{"1min":5.282,"5min":5.282,"15min":5.282},"max":{"1min":57.453000000000003,"5min":286.57400000000001,"15min":286.57400000000001},"last":9.2789999999999999}]},{"osd":3,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":23.404,"5min":20.637,"15min":20.605},"min":{"1min":6.3289999999999997,"5min":6.3289999999999997,"15min":5.9459999999999997},"max":{"1min":167.83799999999999,"5min":167.83799999999999,"15min":177.66},"last":8.5220000000000002},{"interface":"front","average":{"1min":30.977,"5min":28.187999999999999,"15min":31.155999999999999},"min":{"1min":11.647,"5min":8.1829999999999998,"15min":7.609},"max":{"1min":156.17699999999999,"5min":156.17699999999999,"15min":273.99900000000002},"last":18.132999999999999}]},{"osd":4,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":20.108000000000001,"5min":29.747,"15min":26.818000000000001},"min":{"1min":7.7789999999999999,"5min":6.9109999999999996,"15min":6.9109999999999996},"max":{"1min":71.611000000000004,"5min":275.66300000000001,"15min":349.90300000000002},"last":10.746},{"interface":"front","average":{"1min":17.427,"5min":26.164999999999999,"15min":22.431000000000001},"min":{"1min":7.1589999999999998,"5min":7.1589999999999998,"15min":6.6760000000000002},"max":{"1min":73.219999999999999,"5min":285.12700000000001,"15min":285.12700000000001},"last":16.579000000000001}]},{"osd":5,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":22.722999999999999,"5min":24.5,"15min":29.382999999999999},"min":{"1min":11.710000000000001,"5min":9.7840000000000007,"15min":8.0619999999999994},"max":{"1min":89.492000000000004,"5min":138.34899999999999,"15min":353.63400000000001},"last":15.776999999999999},{"interface":"front","average":{"1min":26.366,"5min":30.738,"15min":34.677999999999997},"min":{"1min":14.381,"5min":13.541,"15min":11.539999999999999},"max":{"1min":96.620999999999995,"5min":146.41300000000001,"15min":354.56200000000001},"last":19.760000000000002}]},{"osd":6,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":31.605,"5min":30.536999999999999,"15min":31.172000000000001},"min":{"1min":16.321000000000002,"5min":11.292999999999999,"15min":9.7590000000000003},"max":{"1min":153.01400000000001,"5min":153.01400000000001,"15min":213.11000000000001},"last":21.431000000000001},{"interface":"front","average":{"1min":33.220999999999997,"5min":34.654000000000003,"15min":34.892000000000003},"min":{"1min":15.217000000000001,"5min":12.462999999999999,"15min":12.462999999999999},"max":{"1min":144.51400000000001,"5min":288.30000000000001,"15min":288.30000000000001},"last":20.373000000000001}]},{"osd":7,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":25.690999999999999,"5min":27.300000000000001,"15min":26.981000000000002},"min":{"1min":11.557,"5min":6.375,"15min":6.375},"max":{"1min":90.355999999999995,"5min":216.095,"15min":350.81599999999997},"last":18.914999999999999},{"interface":"front","average":{"1min":21.873000000000001,"5min":22.734000000000002,"15min":23.170000000000002},"min":{"1min":8.6400000000000006,"5min":7.5419999999999998,"15min":7.1669999999999998},"max":{"1min":96.100999999999999,"5min":141.59100000000001,"15min":210.55500000000001},"last":13.491}]}]},{"osd":1,"up_from":16,"seq":68719477001,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67056,"kb_used_data":13672,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94304784,"statfs":{"total":96636764160,"available":96568098816,"internally_reserved":0,"allocated":14000128,"data_stored":8680375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":23.388000000000002,"5min":20.16,"15min":21.632000000000001},"min":{"1min":6.165,"5min":5.9809999999999999,"15min":5.1600000000000001},"max":{"1min":163.62,"5min":221.589,"15min":349.75799999999998},"last":7.4950000000000001},{"interface":"front","average":{"1min":32.435000000000002,"5min":29.16,"15min":26.466999999999999},"min":{"1min":9.4299999999999997,"5min":7.1529999999999996,"15min":6.5309999999999997},"max":{"1min":240.31200000000001,"5min":318.02600000000001,"15min":318.02600000000001},"last":8.1340000000000003}]},{"osd":2,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":28.914000000000001,"5min":20.263000000000002,"15min":22.899999999999999},"min":{"1min":6.7889999999999997,"5min":5.1429999999999998,"15min":4.9619999999999997},"max":{"1min":148.28299999999999,"5min":161.10900000000001,"15min":353.286},"last":8.4269999999999996},{"interface":"front","average":{"1min":34.442,"5min":21.100999999999999,"15min":19.414999999999999},"min":{"1min":7.7750000000000004,"5min":6.3280000000000003,"15min":5.7149999999999999},"max":{"1min":169.84100000000001,"5min":169.84100000000001,"15min":247.99199999999999},"last":10.228999999999999}]},{"osd":3,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":30.649000000000001,"5min":28.231999999999999,"15min":24.68},"min":{"1min":7.0339999999999998,"5min":6.665,"15min":6.5899999999999999},"max":{"1min":169.172,"5min":244.58199999999999,"15min":440.05599999999998},"last":18.545999999999999},{"interface":"front","average":{"1min":31.716000000000001,"5min":30.024000000000001,"15min":28.202999999999999},"min":{"1min":8.4369999999999994,"5min":8.4369999999999994,"15min":5.9889999999999999},"max":{"1min":168.58000000000001,"5min":242.29400000000001,"15min":383.39800000000002},"last":10.875}]},{"osd":4,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":20.422000000000001,"5min":17.603000000000002,"15min":21.190999999999999},"min":{"1min":8.9909999999999997,"5min":6.7830000000000004,"15min":6.5789999999999997},"max":{"1min":63.802999999999997,"5min":113.84,"15min":360.161},"last":13.516},{"interface":"front","average":{"1min":27.309999999999999,"5min":24.93,"15min":23.541},"min":{"1min":9.0510000000000002,"5min":9.0510000000000002,"15min":7.4409999999999998},"max":{"1min":114.33199999999999,"5min":226.68000000000001,"15min":364.39800000000002},"last":12.132}]},{"osd":5,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":39.484999999999999,"5min":28.780000000000001,"15min":29.396999999999998},"min":{"1min":9.6669999999999998,"5min":9.6669999999999998,"15min":7.4359999999999999},"max":{"1min":157.357,"5min":157.357,"15min":282.63900000000001},"last":11.548999999999999},{"interface":"front","average":{"1min":29.167999999999999,"5min":24.120999999999999,"15min":26.896000000000001},"min":{"1min":7.6630000000000003,"5min":7.2889999999999997,"15min":7.1529999999999996},"max":{"1min":105.211,"5min":163.33600000000001,"15min":382.01400000000001},"last":14.728}]},{"osd":6,"last update":"Sat Nov 25 05:26:23 2023","interfaces":[{"interface":"back","average":{"1min":22.631,"5min":23.855,"15min":27.713999999999999},"min":{"1min":9.2360000000000007,"5min":7.5209999999999999,"15min":6.8230000000000004},"max":{"1min":71.406999999999996,"5min":187.62100000000001,"15min":493.18799999999999},"last":14.132999999999999},{"interface":"front","average":{"1min":26.407,"5min":26.518999999999998,"15min":30.838000000000001},"min":{"1min":13.795,"5min":9.9260000000000002,"15min":9.423},"max":{"1min":75.549999999999997,"5min":131.99600000000001,"15min":387.71199999999999},"last":15.439}]},{"osd":7,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":23.481999999999999,"5min":23.608000000000001,"15min":24.239999999999998},"min":{"1min":8.8559999999999999,"5min":7.0279999999999996,"15min":7.0279999999999996},"max":{"1min":116.39400000000001,"5min":219.57900000000001,"15min":272.03699999999998},"last":20.73},{"interface":"front","average":{"1min":30.045999999999999,"5min":26.843,"15min":27.969000000000001},"min":{"1min":11.65,"5min":10.239000000000001,"15min":8.3889999999999993},"max":{"1min":118.447,"5min":126.80200000000001,"15min":297.976},"last":16.937000000000001}]}]},{"osd":3,"up_from":16,"seq":68719477002,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84748,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94287092,"statfs":{"total":96636764160,"available":96549982208,"internally_reserved":0,"allocated":13598720,"data_stored":8282535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":21.867000000000001,"5min":16.334,"15min":21.956},"min":{"1min":5.556,"5min":4.3090000000000002,"15min":4.3090000000000002},"max":{"1min":100.45399999999999,"5min":124.46299999999999,"15min":394.84199999999998},"last":7.6580000000000004},{"interface":"front","average":{"1min":19.355,"5min":17.161000000000001,"15min":21.390999999999998},"min":{"1min":7.2969999999999997,"5min":4.9020000000000001,"15min":4.5890000000000004},"max":{"1min":84.197000000000003,"5min":119.611,"15min":379.51900000000001},"last":11.183}]},{"osd":1,"last update":"Sat Nov 25 05:26:14 2023","interfaces":[{"interface":"back","average":{"1min":21.181999999999999,"5min":20.585000000000001,"15min":28.129999999999999},"min":{"1min":7.6379999999999999,"5min":7.1779999999999999,"15min":5.4080000000000004},"max":{"1min":75.337999999999994,"5min":132.477,"15min":674.39599999999996},"last":9.9969999999999999},{"interface":"front","average":{"1min":18.774000000000001,"5min":24.016999999999999,"15min":26.608000000000001},"min":{"1min":6.1829999999999998,"5min":6.1829999999999998,"15min":6.1829999999999998},"max":{"1min":62.155999999999999,"5min":253.03100000000001,"15min":399.98500000000001},"last":15}]},{"osd":2,"last update":"Sat Nov 25 05:26:23 2023","interfaces":[{"interface":"back","average":{"1min":25.303000000000001,"5min":17.934000000000001,"15min":29.198},"min":{"1min":8.8580000000000005,"5min":7.2649999999999997,"15min":7.0620000000000003},"max":{"1min":84.911000000000001,"5min":84.911000000000001,"15min":673.12400000000002},"last":11.881},{"interface":"front","average":{"1min":26.186,"5min":23.507000000000001,"15min":29.689},"min":{"1min":9.1300000000000008,"5min":7.351,"15min":7.351},"max":{"1min":79.689999999999998,"5min":227.36199999999999,"15min":462.27699999999999},"last":16.471}]},{"osd":4,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":21.428999999999998,"5min":21.751999999999999,"15min":31.024000000000001},"min":{"1min":8.8360000000000003,"5min":8.8360000000000003,"15min":7.6890000000000001},"max":{"1min":81.578000000000003,"5min":141.38399999999999,"15min":676.21299999999997},"last":19.451000000000001},{"interface":"front","average":{"1min":19.236000000000001,"5min":20.013999999999999,"15min":23.495999999999999},"min":{"1min":7.6040000000000001,"5min":6.641,"15min":6.1559999999999997},"max":{"1min":77.034999999999997,"5min":313.18299999999999,"15min":313.18299999999999},"last":12.506}]},{"osd":5,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":26.818000000000001,"5min":23.085999999999999,"15min":32.587000000000003},"min":{"1min":10.710000000000001,"5min":9.6999999999999993,"15min":9.3360000000000003},"max":{"1min":80.661000000000001,"5min":194.51400000000001,"15min":675.37800000000004},"last":20.094000000000001},{"interface":"front","average":{"1min":27.657,"5min":21.199999999999999,"15min":28.454999999999998},"min":{"1min":11.975,"5min":7.5190000000000001,"15min":7.5190000000000001},"max":{"1min":103.021,"5min":131.768,"15min":314.40800000000002},"last":17.809000000000001}]},{"osd":6,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":23.952999999999999,"5min":20.739999999999998,"15min":24.882000000000001},"min":{"1min":8.5519999999999996,"5min":8.3030000000000008,"15min":7.0190000000000001},"max":{"1min":83.287999999999997,"5min":172.66,"15min":312.20699999999999},"last":13.244},{"interface":"front","average":{"1min":26.363,"5min":24.664000000000001,"15min":35.289999999999999},"min":{"1min":11.327,"5min":10.263999999999999,"15min":10.029},"max":{"1min":102.366,"5min":185.809,"15min":677.03499999999997},"last":20.715}]},{"osd":7,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":23.309000000000001,"5min":23.741,"15min":28.478999999999999},"min":{"1min":11.162000000000001,"5min":8.1609999999999996,"15min":7.6420000000000003},"max":{"1min":68.536000000000001,"5min":315.61500000000001,"15min":315.61500000000001},"last":15.786},{"interface":"front","average":{"1min":26.361999999999998,"5min":24.396000000000001,"15min":30.718},"min":{"1min":10.535,"5min":10.535,"15min":10.449999999999999},"max":{"1min":102.361,"5min":145.86099999999999,"15min":320.38400000000001},"last":18.431999999999999}]}]},{"osd":0,"up_from":16,"seq":68719477003,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66664,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305176,"statfs":{"total":96636764160,"available":96568500224,"internally_reserved":0,"allocated":13598720,"data_stored":8282535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":12.904999999999999,"5min":15.397,"15min":17.724},"min":{"1min":4.6100000000000003,"5min":4.6100000000000003,"15min":4.6100000000000003},"max":{"1min":39.119,"5min":109.95399999999999,"15min":209.29900000000001},"last":6.9589999999999996},{"interface":"front","average":{"1min":12.183999999999999,"5min":16.882000000000001,"15min":19.984999999999999},"min":{"1min":5.6219999999999999,"5min":4.5869999999999997,"15min":4.4420000000000002},"max":{"1min":56.362000000000002,"5min":119.73,"15min":401.36000000000001},"last":8.4529999999999994}]},{"osd":2,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":17.683,"5min":19.295000000000002,"15min":22.402999999999999},"min":{"1min":6.6680000000000001,"5min":6.5380000000000003,"15min":6.1260000000000003},"max":{"1min":71.555999999999997,"5min":94.311999999999998,"15min":460.572},"last":11.416},{"interface":"front","average":{"1min":17.940000000000001,"5min":18.923999999999999,"15min":22.27},"min":{"1min":7.4980000000000002,"5min":6.5279999999999996,"15min":6.3090000000000002},"max":{"1min":73.290000000000006,"5min":89.322999999999993,"15min":405.94099999999997},"last":9.1349999999999998}]},{"osd":3,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":21.096,"5min":23.585999999999999,"15min":24.260000000000002},"min":{"1min":7.2999999999999998,"5min":7.2999999999999998,"15min":6.8540000000000001},"max":{"1min":53.616999999999997,"5min":157.65299999999999,"15min":267.74599999999998},"last":17.277999999999999},{"interface":"front","average":{"1min":19.989000000000001,"5min":23.132000000000001,"15min":23.312000000000001},"min":{"1min":9.8480000000000008,"5min":7.1269999999999998,"15min":6.4080000000000004},"max":{"1min":55.122,"5min":128.14500000000001,"15min":266.81900000000002},"last":13.4}]},{"osd":4,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":21.091000000000001,"5min":20.103999999999999,"15min":20.606000000000002},"min":{"1min":9.4649999999999999,"5min":8.3940000000000001,"15min":6.2750000000000004},"max":{"1min":72.540999999999997,"5min":115.369,"15min":147.86099999999999},"last":9.8360000000000003},{"interface":"front","average":{"1min":26.452000000000002,"5min":23.149999999999999,"15min":22.498999999999999},"min":{"1min":9.2189999999999994,"5min":6.8250000000000002,"15min":6.8250000000000002},"max":{"1min":110.93300000000001,"5min":127.291,"15min":148.96199999999999},"last":16.762}]},{"osd":5,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.902999999999999,"5min":23.966999999999999,"15min":23.213999999999999},"min":{"1min":9.0109999999999992,"5min":6.9829999999999997,"15min":6.9829999999999997},"max":{"1min":69.957999999999998,"5min":184.80000000000001,"15min":184.80000000000001},"last":21.690000000000001},{"interface":"front","average":{"1min":20.024000000000001,"5min":23.408999999999999,"15min":25.248000000000001},"min":{"1min":10.836,"5min":7.3849999999999998,"15min":7.3849999999999998},"max":{"1min":58.210000000000001,"5min":131.583,"15min":395.65100000000001},"last":18.824000000000002}]},{"osd":6,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":21.329000000000001,"5min":23.550000000000001,"15min":24.617000000000001},"min":{"1min":9.2829999999999995,"5min":8.4809999999999999,"15min":8.4809999999999999},"max":{"1min":71.225999999999999,"5min":116.63800000000001,"15min":161.24299999999999},"last":19.617999999999999},{"interface":"front","average":{"1min":26.015999999999998,"5min":25.968,"15min":26.870000000000001},"min":{"1min":13.154999999999999,"5min":9.7759999999999998,"15min":9.7759999999999998},"max":{"1min":71.831999999999994,"5min":115.986,"15min":138.46799999999999},"last":23.065999999999999}]},{"osd":7,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.158000000000001,"5min":18.911000000000001,"15min":20.722999999999999},"min":{"1min":10.941000000000001,"5min":8.1340000000000003,"15min":7.1079999999999997},"max":{"1min":40.393999999999998,"5min":112.73099999999999,"15min":140.595},"last":10.441000000000001},{"interface":"front","average":{"1min":21.975999999999999,"5min":22.829999999999998,"15min":25.202999999999999},"min":{"1min":11.441000000000001,"5min":10.888999999999999,"15min":9.9860000000000007},"max":{"1min":57.088000000000001,"5min":103.19,"15min":339.298},"last":20.221}]}]},{"osd":6,"up_from":15,"seq":64424509706,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84736,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94287104,"statfs":{"total":96636764160,"available":96549994496,"internally_reserved":0,"allocated":13598720,"data_stored":8282535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":16.198,"5min":17.219999999999999,"15min":19.709},"min":{"1min":4.7489999999999997,"5min":4.6740000000000004,"15min":4.4409999999999998},"max":{"1min":117.05500000000001,"5min":117.05500000000001,"15min":203.857},"last":8.5419999999999998},{"interface":"front","average":{"1min":15.382999999999999,"5min":16.821000000000002,"15min":20.439},"min":{"1min":5.4820000000000002,"5min":5.1210000000000004,"15min":4.6289999999999996},"max":{"1min":117.795,"5min":236.78700000000001,"15min":381.43900000000002},"last":6.9580000000000002}]},{"osd":1,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.943999999999999,"5min":22.879000000000001,"15min":26.055},"min":{"1min":8.0890000000000004,"5min":6.2839999999999998,"15min":6.165},"max":{"1min":143.61099999999999,"5min":229.53399999999999,"15min":310.423},"last":13.673999999999999},{"interface":"front","average":{"1min":20.885999999999999,"5min":20.399999999999999,"15min":26.382999999999999},"min":{"1min":6.3360000000000003,"5min":6.2030000000000003,"15min":5.1310000000000002},"max":{"1min":190.029,"5min":243.74700000000001,"15min":548.03599999999994},"last":8.1159999999999997}]},{"osd":2,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":28.033999999999999,"5min":22.832000000000001,"15min":27.013999999999999},"min":{"1min":7.9550000000000001,"5min":6.9690000000000003,"15min":6.9690000000000003},"max":{"1min":122.754,"5min":251.524,"15min":353.80399999999997},"last":14.712},{"interface":"front","average":{"1min":26.027000000000001,"5min":27.611999999999998,"15min":30.841999999999999},"min":{"1min":10.65,"5min":10.65,"15min":10.617000000000001},"max":{"1min":120.961,"5min":249.97399999999999,"15min":304.49900000000002},"last":15.483000000000001}]},{"osd":3,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":25.802,"5min":31.065999999999999,"15min":32.701999999999998},"min":{"1min":11.24,"5min":10.164,"15min":7.5419999999999998},"max":{"1min":185.28299999999999,"5min":259.89299999999997,"15min":295.82799999999997},"last":16.358000000000001},{"interface":"front","average":{"1min":22.247,"5min":26.661000000000001,"15min":26.797999999999998},"min":{"1min":7.7720000000000002,"5min":6.9199999999999999,"15min":6.0209999999999999},"max":{"1min":168.56100000000001,"5min":249.07400000000001,"15min":266.40199999999999},"last":10.927}]},{"osd":4,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":31.027999999999999,"5min":31.033000000000001,"15min":32.439999999999998},"min":{"1min":10.000999999999999,"5min":10.000999999999999,"15min":10.000999999999999},"max":{"1min":186.28,"5min":277.85000000000002,"15min":330.86900000000003},"last":17.282},{"interface":"front","average":{"1min":25.614999999999998,"5min":26.050000000000001,"15min":25.638000000000002},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":6.5919999999999996},"max":{"1min":192.25999999999999,"5min":288.13099999999997,"15min":288.13099999999997},"last":9.3320000000000007}]},{"osd":5,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.140000000000001,"5min":18.035,"15min":22.939},"min":{"1min":6.0039999999999996,"5min":5.5590000000000002,"15min":5.5590000000000002},"max":{"1min":121.31399999999999,"5min":237.46100000000001,"15min":281.13999999999999},"last":7.7450000000000001},{"interface":"front","average":{"1min":25.876000000000001,"5min":28.905999999999999,"15min":33.758000000000003},"min":{"1min":8.0069999999999997,"5min":8.0069999999999997,"15min":8.0069999999999997},"max":{"1min":133.44999999999999,"5min":269.77100000000002,"15min":311.09199999999998},"last":18.093}]},{"osd":7,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":29.199999999999999,"5min":31.532,"15min":34.887999999999998},"min":{"1min":13.699,"5min":12.319000000000001,"15min":9.2210000000000001},"max":{"1min":140.25200000000001,"5min":259.87700000000001,"15min":367.24900000000002},"last":18.664999999999999},{"interface":"front","average":{"1min":25.48,"5min":24.911999999999999,"15min":28.138999999999999},"min":{"1min":8.8529999999999998,"5min":8.3469999999999995,"15min":7.7199999999999998},"max":{"1min":161.37899999999999,"5min":269.14400000000001,"15min":325.54199999999997},"last":12.162000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509708,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70840,"kb_used_data":13284,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94301000,"statfs":{"total":96636764160,"available":96564224000,"internally_reserved":0,"allocated":13602816,"data_stored":8282540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:26 2023","interfaces":[{"interface":"back","average":{"1min":18.672999999999998,"5min":18.614000000000001,"15min":19.898},"min":{"1min":4.8550000000000004,"5min":4.3170000000000002,"15min":3.8279999999999998},"max":{"1min":124.38200000000001,"5min":150.73599999999999,"15min":219.172},"last":14.093999999999999},{"interface":"front","average":{"1min":19.463999999999999,"5min":17.724,"15min":18.888000000000002},"min":{"1min":4.9299999999999997,"5min":4.5369999999999999,"15min":4.1289999999999996},"max":{"1min":204.953,"5min":216.86600000000001,"15min":216.86600000000001},"last":6.7619999999999996}]},{"osd":1,"last update":"Sat Nov 25 05:26:26 2023","interfaces":[{"interface":"back","average":{"1min":15.097,"5min":18.919,"15min":18.690000000000001},"min":{"1min":5.79,"5min":5.2149999999999999,"15min":4.9749999999999996},"max":{"1min":89.542000000000002,"5min":205.36500000000001,"15min":205.36500000000001},"last":9.3390000000000004},{"interface":"front","average":{"1min":25.373000000000001,"5min":21.013999999999999,"15min":20.709},"min":{"1min":8.6920000000000002,"5min":6.5759999999999996,"15min":6.5759999999999996},"max":{"1min":196.667,"5min":196.667,"15min":196.667},"last":8.6920000000000002}]},{"osd":2,"last update":"Sat Nov 25 05:26:26 2023","interfaces":[{"interface":"back","average":{"1min":28.908000000000001,"5min":21.404,"15min":23.119},"min":{"1min":8.8520000000000003,"5min":6.9320000000000004,"15min":6.9320000000000004},"max":{"1min":277.51299999999998,"5min":277.51299999999998,"15min":277.51299999999998},"last":13.077999999999999},{"interface":"front","average":{"1min":31.375,"5min":24.786999999999999,"15min":26.934000000000001},"min":{"1min":14.608000000000001,"5min":10.664,"15min":8.5899999999999999},"max":{"1min":256.63,"5min":256.63,"15min":312.05200000000002},"last":15.534000000000001}]},{"osd":3,"last update":"Sat Nov 25 05:26:26 2023","interfaces":[{"interface":"back","average":{"1min":25.312000000000001,"5min":23.844000000000001,"15min":26.222999999999999},"min":{"1min":6.8760000000000003,"5min":6.8760000000000003,"15min":6.133},"max":{"1min":210.71299999999999,"5min":210.71299999999999,"15min":359.23700000000002},"last":11.175000000000001},{"interface":"front","average":{"1min":21.283999999999999,"5min":24.105,"15min":23.407},"min":{"1min":6.1719999999999997,"5min":6.1719999999999997,"15min":6.1719999999999997},"max":{"1min":137.21199999999999,"5min":196.55000000000001,"15min":381.53100000000001},"last":18.224}]},{"osd":5,"last update":"Sat Nov 25 05:25:56 2023","interfaces":[{"interface":"back","average":{"1min":18.43,"5min":20.623999999999999,"15min":26.562000000000001},"min":{"1min":5.875,"5min":5.875,"15min":5.875},"max":{"1min":72.236999999999995,"5min":195.851,"15min":360.387},"last":7.9130000000000003},{"interface":"front","average":{"1min":24.617000000000001,"5min":24.105,"15min":29.986999999999998},"min":{"1min":12.101000000000001,"5min":9.4469999999999992,"15min":9.4469999999999992},"max":{"1min":122.352,"5min":159.67099999999999,"15min":274.38200000000001},"last":14.944000000000001}]},{"osd":6,"last update":"Sat Nov 25 05:25:56 2023","interfaces":[{"interface":"back","average":{"1min":20.521000000000001,"5min":24.015000000000001,"15min":27.704999999999998},"min":{"1min":10.093,"5min":10.093,"15min":9.8409999999999993},"max":{"1min":48.161000000000001,"5min":169.44499999999999,"15min":181.43299999999999},"last":16.16},{"interface":"front","average":{"1min":24.329000000000001,"5min":22.428999999999998,"15min":23.622},"min":{"1min":15.603999999999999,"5min":7.1619999999999999,"15min":5.6390000000000002},"max":{"1min":60.040999999999997,"5min":212.25399999999999,"15min":219.26499999999999},"last":12.365}]},{"osd":7,"last update":"Sat Nov 25 05:25:56 2023","interfaces":[{"interface":"back","average":{"1min":14.73,"5min":23.181000000000001,"15min":25.620999999999999},"min":{"1min":8.7100000000000009,"5min":7.0369999999999999,"15min":7.0369999999999999},"max":{"1min":27.728999999999999,"5min":202.363,"15min":215.512},"last":17.434000000000001},{"interface":"front","average":{"1min":14.417999999999999,"5min":20.943999999999999,"15min":24.698},"min":{"1min":6.0350000000000001,"5min":6.0350000000000001,"15min":6.0350000000000001},"max":{"1min":29.920999999999999,"5min":211.62200000000001,"15min":248.83199999999999},"last":10.488}]}]},{"osd":7,"up_from":13,"seq":55834575114,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66692,"kb_used_data":13284,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305148,"statfs":{"total":96636764160,"available":96568471552,"internally_reserved":0,"allocated":13602816,"data_stored":8282540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":11.608000000000001,"5min":13.819000000000001,"15min":17.303999999999998},"min":{"1min":4.6150000000000002,"5min":4.3579999999999997,"15min":4.3579999999999997},"max":{"1min":23.535,"5min":97.569000000000003,"15min":222.50800000000001},"last":6.8460000000000001},{"interface":"front","average":{"1min":9.8520000000000003,"5min":15.375999999999999,"15min":18.193999999999999},"min":{"1min":4.7450000000000001,"5min":4.6600000000000001,"15min":4.5579999999999998},"max":{"1min":25.248000000000001,"5min":218.03999999999999,"15min":218.03999999999999},"last":4.8280000000000003}]},{"osd":1,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.643000000000001,"5min":13.715999999999999,"15min":17.489000000000001},"min":{"1min":5.4859999999999998,"5min":5.4859999999999998,"15min":5.2619999999999996},"max":{"1min":79.078999999999994,"5min":79.078999999999994,"15min":165.99000000000001},"last":8.0679999999999996},{"interface":"front","average":{"1min":20.498000000000001,"5min":18.827000000000002,"15min":22.771999999999998},"min":{"1min":7.508,"5min":6.8140000000000001,"15min":6.7030000000000003},"max":{"1min":117.25,"5min":117.25,"15min":303.892},"last":14.558}]},{"osd":2,"last update":"Sat Nov 25 05:26:18 2023","interfaces":[{"interface":"back","average":{"1min":12.206,"5min":18.088999999999999,"15min":20.989000000000001},"min":{"1min":6.3810000000000002,"5min":6.3810000000000002,"15min":6.3810000000000002},"max":{"1min":23.623999999999999,"5min":169.262,"15min":186.08600000000001},"last":9.2170000000000005},{"interface":"front","average":{"1min":16.013000000000002,"5min":16.907,"15min":21.454000000000001},"min":{"1min":9.0570000000000004,"5min":7.508,"15min":6.4279999999999999},"max":{"1min":27.609999999999999,"5min":108.26600000000001,"15min":337.41899999999998},"last":15.180999999999999}]},{"osd":3,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":16.559999999999999,"5min":22.437999999999999,"15min":23.844999999999999},"min":{"1min":12.122999999999999,"5min":8.4469999999999992,"15min":7.5199999999999996},"max":{"1min":28.463000000000001,"5min":152.47399999999999,"15min":238.941},"last":16.091999999999999},{"interface":"front","average":{"1min":18.123000000000001,"5min":21.559999999999999,"15min":26.684999999999999},"min":{"1min":12.221,"5min":6.6929999999999996,"15min":5.5949999999999998},"max":{"1min":29.536000000000001,"5min":150.208,"15min":1292.413},"last":16.710000000000001}]},{"osd":4,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.506,"5min":23.238,"15min":25.268000000000001},"min":{"1min":8.8109999999999999,"5min":8.8109999999999999,"15min":7.9340000000000002},"max":{"1min":21.449999999999999,"5min":120.414,"15min":328.31400000000002},"last":12.369},{"interface":"front","average":{"1min":27.879000000000001,"5min":21.405000000000001,"15min":23.079999999999998},"min":{"1min":10.739000000000001,"5min":7.1849999999999996,"15min":6.4320000000000004},"max":{"1min":192.684,"5min":192.684,"15min":302.97199999999998},"last":17.718}]},{"osd":5,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":17.332000000000001,"5min":22.279,"15min":24.291},"min":{"1min":9.1509999999999998,"5min":7.7919999999999998,"15min":7.7619999999999996},"max":{"1min":42.118000000000002,"5min":186.69499999999999,"15min":217.023},"last":13.510999999999999},{"interface":"front","average":{"1min":23.495000000000001,"5min":28.128,"15min":29.045999999999999},"min":{"1min":11.694000000000001,"5min":10.643000000000001,"15min":7.1130000000000004},"max":{"1min":83.798000000000002,"5min":210.22300000000001,"15min":318.11599999999999},"last":18.609999999999999}]},{"osd":6,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.75,"5min":20.798999999999999,"15min":22.933},"min":{"1min":7.069,"5min":6.2380000000000004,"15min":6.2380000000000004},"max":{"1min":100.828,"5min":188.15199999999999,"15min":282.28100000000001},"last":10.057},{"interface":"front","average":{"1min":18.027999999999999,"5min":23.329000000000001,"15min":25.382000000000001},"min":{"1min":8.0459999999999994,"5min":8.0459999999999994,"15min":8.0459999999999994},"max":{"1min":100.143,"5min":187.09899999999999,"15min":215.00700000000001},"last":10.74}]}]},{"osd":5,"up_from":13,"seq":55834575114,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89264,"kb_used_data":13672,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94282576,"statfs":{"total":96636764160,"available":96545357824,"internally_reserved":0,"allocated":14000128,"data_stored":8680375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":17.623000000000001,"5min":14.541,"15min":18.605},"min":{"1min":4.5910000000000002,"5min":4.202,"15min":4.202},"max":{"1min":133.92599999999999,"5min":167.83600000000001,"15min":471.81},"last":15.395},{"interface":"front","average":{"1min":14.865,"5min":15.755000000000001,"15min":20.606999999999999},"min":{"1min":6.2699999999999996,"5min":5.4279999999999999,"15min":4.5529999999999999},"max":{"1min":46.905000000000001,"5min":197.834,"15min":317.45699999999999},"last":18.481000000000002}]},{"osd":1,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":27.741,"5min":18.356000000000002,"15min":21.565000000000001},"min":{"1min":6.3470000000000004,"5min":5.4189999999999996,"15min":5.0709999999999997},"max":{"1min":140.28999999999999,"5min":140.28999999999999,"15min":263.589},"last":10.336},{"interface":"front","average":{"1min":30.024000000000001,"5min":17.751000000000001,"15min":23.059999999999999},"min":{"1min":6.351,"5min":5.1459999999999999,"15min":5.1459999999999999},"max":{"1min":150.196,"5min":150.196,"15min":238.26499999999999},"last":9.7850000000000001}]},{"osd":2,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":19.062999999999999,"5min":19.102,"15min":23.210000000000001},"min":{"1min":7.1890000000000001,"5min":6.2240000000000002,"15min":6.2240000000000002},"max":{"1min":82.295000000000002,"5min":182.477,"15min":416.83199999999999},"last":8.2599999999999998},{"interface":"front","average":{"1min":28.707000000000001,"5min":25.138000000000002,"15min":25.091999999999999},"min":{"1min":12.006,"5min":9.4000000000000004,"15min":7.0880000000000001},"max":{"1min":179.749,"5min":190.517,"15min":207.28700000000001},"last":10.734}]},{"osd":3,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":31.044,"5min":24.114000000000001,"15min":29.751999999999999},"min":{"1min":10.327,"5min":10.097,"15min":6.6079999999999997},"max":{"1min":166.59700000000001,"5min":166.59700000000001,"15min":443.10300000000001},"last":14.766},{"interface":"front","average":{"1min":28.024999999999999,"5min":19.308,"15min":31.902000000000001},"min":{"1min":7.851,"5min":6.9800000000000004,"15min":6.2199999999999998},"max":{"1min":130.87299999999999,"5min":130.87299999999999,"15min":1666.6489999999999},"last":7.569}]},{"osd":4,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":30.039000000000001,"5min":25.219999999999999,"15min":25.515999999999998},"min":{"1min":10.957000000000001,"5min":10.657,"15min":9.0280000000000005},"max":{"1min":173.041,"5min":173.041,"15min":173.041},"last":17.670000000000002},{"interface":"front","average":{"1min":17.225999999999999,"5min":17.971,"15min":21.449000000000002},"min":{"1min":6.5330000000000004,"5min":6.5330000000000004,"15min":6.5330000000000004},"max":{"1min":68.790000000000006,"5min":123.01300000000001,"15min":156.92500000000001},"last":13.939}]},{"osd":6,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":26.797999999999998,"5min":26.663,"15min":31.244},"min":{"1min":13.439,"5min":11.666,"15min":9.9979999999999993},"max":{"1min":100.47799999999999,"5min":229.86199999999999,"15min":229.86199999999999},"last":11.395},{"interface":"front","average":{"1min":31.841999999999999,"5min":24.629000000000001,"15min":27.375},"min":{"1min":8.6280000000000001,"5min":8.1159999999999997,"15min":7.4770000000000003},"max":{"1min":165.46000000000001,"5min":234.31100000000001,"15min":234.31100000000001},"last":12.688000000000001}]},{"osd":7,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":25.68,"5min":20.594999999999999,"15min":26.533999999999999},"min":{"1min":7.2679999999999998,"5min":6.2370000000000001,"15min":6.2370000000000001},"max":{"1min":67.870999999999995,"5min":195.94399999999999,"15min":195.94399999999999},"last":16.085999999999999},{"interface":"front","average":{"1min":23.308,"5min":25.666,"15min":26.741},"min":{"1min":11.151,"5min":11.151,"15min":7.3289999999999997},"max":{"1min":57.329000000000001,"5min":185.74700000000001,"15min":224.99600000000001},"last":13.343999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-25T05:26:28.059 INFO:tasks.ceph:pgid 9.15 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.059 INFO:tasks.ceph:pgid 9.14 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.059 INFO:tasks.ceph:pgid 9.17 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.059 INFO:tasks.ceph:pgid 9.11 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.059 INFO:tasks.ceph:pgid 9.10 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.059 INFO:tasks.ceph:pgid 9.12 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.059 INFO:tasks.ceph:pgid 9.1d last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.060 INFO:tasks.ceph:pgid 9.1c last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.060 INFO:tasks.ceph:pgid 9.1f last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.060 INFO:tasks.ceph:pgid 9.1e last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.060 INFO:tasks.ceph:pgid 9.c last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.060 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-11-25T05:04:31.491894+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=4, tm_sec=31, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.060 INFO:tasks.ceph:pgid 9.d last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.060 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-11-25T05:04:31.491894+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=4, tm_sec=31, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.060 INFO:tasks.ceph:pgid 9.e last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.060 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-11-25T05:04:31.491894+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=4, tm_sec=31, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.061 INFO:tasks.ceph:pgid 9.f last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.061 INFO:tasks.ceph:pgid 9.9 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.061 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-11-25T05:04:31.491894+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=4, tm_sec=31, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.061 INFO:tasks.ceph:pgid 9.a last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.061 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-11-25T05:04:31.491894+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=4, tm_sec=31, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.061 INFO:tasks.ceph:pgid 9.b last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.061 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-11-25T05:04:31.491894+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=4, tm_sec=31, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.061 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-11-25T05:04:31.491894+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=4, tm_sec=31, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.062 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.062 INFO:tasks.ceph:pgid 9.0 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.062 INFO:tasks.ceph:pgid 9.1 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.062 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.062 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.062 INFO:tasks.ceph:pgid 9.5 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.062 INFO:tasks.ceph:pgid 9.1a last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.062 INFO:tasks.ceph:pgid 9.1b last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.063 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.063 INFO:tasks.ceph:pgid 9.19 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:28.063 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-11-25T05:26:31.457 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:13.655 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:31.461 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:13.659 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:31.474 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:13.672 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:31.506 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:13.651 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:31.508 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:13.653 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:31.511 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:13.710 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:31.520 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:13.718 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:31.523 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:13.667 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:31.533 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:13.731 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:36.471 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:18.670 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:36.477 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:18.676 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:36.489 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:18.687 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:36.521 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:18.665 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:36.523 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:18.667 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:36.524 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:18.722 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:36.527 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:18.725 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:36.536 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:18.680 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:36.540 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:18.738 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:46.526 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:28.725 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:46.530 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:28.728 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:46.544 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:28.742 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:46.571 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:28.715 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:46.573 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:28.717 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:46.573 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:28.771 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:46.576 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:28.774 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:46.587 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:28.732 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:46.597 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:28.795 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:48.064 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-25T05:26:48.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.168+0000 7f959496c640 10 monclient: get_monmap_and_config 2023-11-25T05:26:48.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.168+0000 7f959496c640 10 monclient: build_initial_monmap 2023-11-25T05:26:48.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.168+0000 7f959496c640 10 monclient: monmap: 2023-11-25T05:26:48.171 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:48.171 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:48.171 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:26:48.170484+0000 2023-11-25T05:26:48.171 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:26:48.170484+0000 2023-11-25T05:26:48.171 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:48.171 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:48.171 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:26:48.171 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:26:48.171 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:26:48.172 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:26:48.172 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:26:48.172 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.169+0000 7f959496c640 1 Processor -- start 2023-11-25T05:26:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.169+0000 7f959496c640 1 -- start start 2023-11-25T05:26:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.169+0000 7f959496c640 10 monclient: init 2023-11-25T05:26:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.169+0000 7f959496c640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.169+0000 7f959496c640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-11-25T05:26:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.169+0000 7f959496c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f95900b1b70 addr v1:172.21.15.7:6789/0 2023-11-25T05:26:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.169+0000 7f959496c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9590072890 addr v1:172.21.15.90:6789/0 2023-11-25T05:26:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.169+0000 7f959496c640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f959013f7d0 0x7f95900acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.169+0000 7f959496c640 10 monclient(hunting): picked mon.noname-c con 0x7f959013f7d0 addr v2:172.21.15.90:3300/0 2023-11-25T05:26:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.170+0000 7f959496c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95900ad520 con 0x7f95900b1b70 2023-11-25T05:26:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.170+0000 7f959496c640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95900ae1f0 con 0x7f9590072890 2023-11-25T05:26:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.170+0000 7f959496c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.170+0000 7f959496c640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95900ad930 con 0x7f959013f7d0 2023-11-25T05:26:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.170+0000 7f959496c640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.170+0000 7f959496c640 10 monclient(hunting): authenticate will time out at 2734.413852s 2023-11-25T05:26:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.170+0000 7f958e59c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f95900b1b70 0x7f95901453c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36658/0 (socket says 172.21.15.7:36658) 2023-11-25T05:26:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.170+0000 7f958e59c640 1 -- 172.21.15.7:0/1507394014 learned_addr learned my addr 172.21.15.7:0/1507394014 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:26:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.170+0000 7f958ed9d640 1 --2- 172.21.15.7:0/1507394014 >> v2:172.21.15.90:3300/0 conn(0x7f959013f7d0 0x7f95900acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.171+0000 7f958ed9d640 10 monclient(hunting): get_auth_request con 0x7f959013f7d0 auth_method 0 2023-11-25T05:26:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.171+0000 7f958ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.171+0000 7f958ed9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.171+0000 7f958ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.171+0000 7f958ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.171+0000 7f958ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.171+0000 7f958ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.172+0000 7f958dd9b640 1 -- 172.21.15.7:0/1507394014 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83569695 0 0) 0x7f95900ad520 con 0x7f95900b1b70 2023-11-25T05:26:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.172+0000 7f958dd9b640 10 monclient(hunting): _negotiate 2023-11-25T05:26:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.172+0000 7f958dd9b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.172+0000 7f958dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.172+0000 7f958dd9b640 10 monclient(hunting): my global_id is 15987 2023-11-25T05:26:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.172+0000 7f958dd9b640 1 -- 172.21.15.7:0/1507394014 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f957c003520 con 0x7f95900b1b70 2023-11-25T05:26:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.173+0000 7f958ed9d640 10 monclient(hunting): handle_auth_done global_id 21313 payload 274 2023-11-25T05:26:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.173+0000 7f958ed9d640 1 -- 172.21.15.7:0/1507394014 >> v1:172.21.15.90:6789/0 conn(0x7f9590072890 legacy=0x7f959013f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.173+0000 7f958ed9d640 1 -- 172.21.15.7:0/1507394014 >> v1:172.21.15.7:6789/0 conn(0x7f95900b1b70 legacy=0x7f95901453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.173+0000 7f958ed9d640 10 monclient: _finish_hunting 0 2023-11-25T05:26:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.173+0000 7f958ed9d640 1 monclient: found mon.noname-c 2023-11-25T05:26:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.173+0000 7f958ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.173+0000 7f958ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.90:3300/0 2023-11-25T05:26:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.173+0000 7f958ed9d640 1 -- 172.21.15.7:0/1507394014 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95900ae4d0 con 0x7f959013f7d0 2023-11-25T05:26:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.175+0000 7f958ed9d640 1 --2- 172.21.15.7:0/1507394014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f959013f7d0 0x7f95900acfe0 secure :-1 s=READY pgs=5175 cs=0 l=1 rev1=1 crypto rx=0x7f9588009d90 tx=0x7f958802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e80f0a42c69cbee4 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f958dd9b640 1 -- 172.21.15.7:0/1507394014 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958802cc60 con 0x7f959013f7d0 2023-11-25T05:26:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f958dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f958dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-25T05:26:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f958dd9b640 10 monclient: dump: 2023-11-25T05:26:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:48.184 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f958dd9b640 10 monclient: _finish_auth 0 2023-11-25T05:26:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f958dd9b640 10 monclient: _check_auth_tickets 2023-11-25T05:26:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f958dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f958dd9b640 1 -- 172.21.15.7:0/1507394014 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958802cdc0 con 0x7f959013f7d0 2023-11-25T05:26:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f959496c640 5 monclient: authenticate success, global_id 21313 2023-11-25T05:26:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f959496c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-25T05:26:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f958dd9b640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f959496c640 10 monclient: get_monmap_and_config success 2023-11-25T05:26:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f958dd9b640 1 -- 172.21.15.7:0/1507394014 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9588034660 con 0x7f959013f7d0 2023-11-25T05:26:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f958dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f959496c640 4 set_mon_vals no callback set 2023-11-25T05:26:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f958dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f958dd9b640 10 monclient: dump: 2023-11-25T05:26:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:48.186 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f959496c640 10 monclient: shutdown 2023-11-25T05:26:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f959496c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f959496c640 1 -- 172.21.15.7:0/1507394014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f959013f7d0 msgr2=0x7f95900acfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f959496c640 1 --2- 172.21.15.7:0/1507394014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f959013f7d0 0x7f95900acfe0 secure :-1 s=READY pgs=5175 cs=0 l=1 rev1=1 crypto rx=0x7f9588009d90 tx=0x7f958802beb0 comp rx=0 tx=0).stop 2023-11-25T05:26:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f959496c640 1 -- 172.21.15.7:0/1507394014 shutdown_connections 2023-11-25T05:26:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f959496c640 1 --2- 172.21.15.7:0/1507394014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f959013f7d0 0x7f95900acfe0 unknown :-1 s=CLOSED pgs=5175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f959496c640 1 -- 172.21.15.7:0/1507394014 >> 172.21.15.7:0/1507394014 conn(0x7f95900b4d20 msgr2=0x7f95900b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f959496c640 1 -- 172.21.15.7:0/1507394014 shutdown_connections 2023-11-25T05:26:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.180+0000 7f959496c640 1 -- 172.21.15.7:0/1507394014 wait complete. 2023-11-25T05:26:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.181+0000 7f959496c640 10 monclient: build_initial_monmap 2023-11-25T05:26:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.181+0000 7f959496c640 10 monclient: monmap: 2023-11-25T05:26:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:26:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:26:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:26:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:26:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:26:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:26:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:26:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:26:48.190 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.181+0000 7f959496c640 1 librados: starting msgr at 2023-11-25T05:26:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.181+0000 7f959496c640 1 librados: starting objecter 2023-11-25T05:26:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.181+0000 7f959496c640 1 Processor -- start 2023-11-25T05:26:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.181+0000 7f959496c640 1 -- start start 2023-11-25T05:26:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.181+0000 7f959496c640 1 librados: setting wanted keys 2023-11-25T05:26:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.181+0000 7f959496c640 1 librados: calling monclient init 2023-11-25T05:26:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.181+0000 7f959496c640 10 monclient: init 2023-11-25T05:26:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 10 monclient: _reopen_session rank -1 2023-11-25T05:26:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-25T05:26:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9590072890 0x7f9590070e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 10 monclient(hunting): picked mon.noname-a con 0x7f9590072890 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:26:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95900b1b70 0x7f95901350d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 10 monclient(hunting): picked mon.noname-c con 0x7f95900b1b70 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:26:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f959013f7d0 0x7f9590132ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 10 monclient(hunting): picked mon.noname-b con 0x7f959013f7d0 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:26:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9590135610 con 0x7f9590072890 2023-11-25T05:26:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9590071990 con 0x7f959013f7d0 2023-11-25T05:26:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 10 monclient(hunting): start opening mon connection 2023-11-25T05:26:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9590135d50 con 0x7f95900b1b70 2023-11-25T05:26:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 10 monclient(hunting): _renew_subs 2023-11-25T05:26:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.182+0000 7f959496c640 10 monclient(hunting): authenticate will time out at 2734.426343s 2023-11-25T05:26:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f9587fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95900b1b70 0x7f95901350d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f958e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9590072890 0x7f9590070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f958ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f959013f7d0 0x7f9590132ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f9587fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95900b1b70 0x7f95901350d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:54120/0 (socket says 172.21.15.7:54120) 2023-11-25T05:26:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f9587fff640 1 -- 172.21.15.7:0/1815284713 learned_addr learned my addr 172.21.15.7:0/1815284713 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:26:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f9587fff640 10 monclient(hunting): get_auth_request con 0x7f95900b1b70 auth_method 0 2023-11-25T05:26:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f9587fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f9587fff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f9587fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f958e59c640 10 monclient(hunting): get_auth_request con 0x7f9590072890 auth_method 0 2023-11-25T05:26:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f958e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f958e59c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f958e59c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f958ed9d640 10 monclient(hunting): get_auth_request con 0x7f959013f7d0 auth_method 0 2023-11-25T05:26:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f958ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:26:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f958ed9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:26:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f958ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:26:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f9587fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f9587fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.183+0000 7f9587fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.184+0000 7f958ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.184+0000 7f958ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.184+0000 7f958ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.184+0000 7f958e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:26:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.184+0000 7f958e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:26:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.184+0000 7f958e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:26:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.186+0000 7f958ed9d640 10 monclient(hunting): handle_auth_done global_id 21316 payload 824 2023-11-25T05:26:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.186+0000 7f958ed9d640 1 -- 172.21.15.7:0/1815284713 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95900b1b70 msgr2=0x7f95901350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.186+0000 7f958ed9d640 1 --2- 172.21.15.7:0/1815284713 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95900b1b70 0x7f95901350d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.186+0000 7f958ed9d640 1 -- 172.21.15.7:0/1815284713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9590072890 msgr2=0x7f9590070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.186+0000 7f958ed9d640 1 --2- 172.21.15.7:0/1815284713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9590072890 0x7f9590070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.186+0000 7f958ed9d640 10 monclient: _finish_hunting 0 2023-11-25T05:26:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.186+0000 7f958ed9d640 1 monclient: found mon.noname-b 2023-11-25T05:26:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.186+0000 7f958ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:26:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.186+0000 7f958ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:26:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.186+0000 7f958ed9d640 1 -- 172.21.15.7:0/1815284713 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9588009a70 con 0x7f959013f7d0 2023-11-25T05:26:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.188+0000 7f958ed9d640 1 --2- 172.21.15.7:0/1815284713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f959013f7d0 0x7f9590132ab0 secure :-1 s=READY pgs=5176 cs=0 l=1 rev1=1 crypto rx=0x7f95900b57b0 tx=0x7f9588004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.188+0000 7f9586ffd640 10 client.?.objecter ms_handle_connect 0x7f959013f7d0 2023-11-25T05:26:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.188+0000 7f9586ffd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:26:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f9586ffd640 1 -- 172.21.15.7:0/1815284713 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958803e610 con 0x7f959013f7d0 2023-11-25T05:26:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f9586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f9586ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:26:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f9586ffd640 10 monclient: dump: 2023-11-25T05:26:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:48.198 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f9586ffd640 10 monclient: _finish_auth 0 2023-11-25T05:26:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f9586ffd640 10 monclient: _check_auth_tickets 2023-11-25T05:26:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f9586ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:26:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f959496c640 5 monclient: authenticate success, global_id 21316 2023-11-25T05:26:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f9586ffd640 1 -- 172.21.15.7:0/1815284713 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958803e770 con 0x7f959013f7d0 2023-11-25T05:26:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f9586ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:26:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f9586ffd640 1 -- 172.21.15.7:0/1815284713 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958803eb20 con 0x7f959013f7d0 2023-11-25T05:26:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f959496c640 10 monclient: _renew_subs 2023-11-25T05:26:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f959496c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f959496c640 1 -- 172.21.15.7:0/1815284713 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9590135830 con 0x7f959013f7d0 2023-11-25T05:26:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f9586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:26:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f9586ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:26:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f9586ffd640 10 monclient: dump: 2023-11-25T05:26:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:26:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:26:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:26:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:26:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:26:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:26:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:26:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:26:48.200 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:26:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f958cd99640 4 set_mon_vals no callback set 2023-11-25T05:26:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f959496c640 10 client.21316.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:26:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f959496c640 10 monclient: _renew_subs 2023-11-25T05:26:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f959496c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f959496c640 1 -- 172.21.15.7:0/1815284713 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9590132ff0 con 0x7f959013f7d0 2023-11-25T05:26:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.194+0000 7f959496c640 1 librados: init done 2023-11-25T05:26:48.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.195+0000 7f959496c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:48.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.195+0000 7f959496c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:48.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.195+0000 7f959496c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:26:48.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.195+0000 7f959496c640 1 -- 172.21.15.7:0/1815284713 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9590075b60 con 0x7f959013f7d0 2023-11-25T05:26:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.213+0000 7f9586ffd640 1 -- 172.21.15.7:0/1815284713 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f958803b050 con 0x7f959013f7d0 2023-11-25T05:26:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.213+0000 7f9586ffd640 1 --2- 172.21.15.7:0/1815284713 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f957006c820 0x7f957006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:26:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.213+0000 7f958e59c640 1 --2- 172.21.15.7:0/1815284713 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f957006c820 0x7f957006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:26:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.213+0000 7f958e59c640 10 monclient: get_auth_request con 0x7f957006c820 auth_method 0 2023-11-25T05:26:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.214+0000 7f958e59c640 1 --2- 172.21.15.7:0/1815284713 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f957006c820 0x7f957006ece0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9580005880 tx=0x7f9580009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:26:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.214+0000 7f9586ffd640 10 client.21316.objecter ms_handle_connect 0x7f957006c820 2023-11-25T05:26:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.217+0000 7f9586ffd640 1 -- 172.21.15.7:0/1815284713 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 687..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f958807b0e0 con 0x7f959013f7d0 2023-11-25T05:26:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.217+0000 7f9586ffd640 10 client.21316.objecter ms_dispatch 0x7f9590000b90 osd_map(918..918 src has 687..918) v4 2023-11-25T05:26:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.217+0000 7f9586ffd640 3 client.21316.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:26:48.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.217+0000 7f9586ffd640 3 client.21316.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:26:48.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.217+0000 7f9586ffd640 20 client.21316.objecter dump_active .. 0 homeless 2023-11-25T05:26:48.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.449+0000 7f9586ffd640 1 -- 172.21.15.7:0/1815284713 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9588077420 con 0x7f959013f7d0 2023-11-25T05:26:48.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.449+0000 7f9586ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:26:48.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.449+0000 7f9586ffd640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:26:48.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.627+0000 7f959496c640 1 -- 172.21.15.7:0/1815284713 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95900b1f50 con 0x7f957006c820 2023-11-25T05:26:48.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.633+0000 7f9586ffd640 1 -- 172.21.15.7:0/1815284713 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152460 (secure 0 0 0) 0x7f958000f820 con 0x7f957006c820 2023-11-25T05:26:48.636 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:26:48.636 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-25T05:26:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.634+0000 7f959496c640 10 librados: watch_flush enter 2023-11-25T05:26:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.634+0000 7f959496c640 10 librados: watch_flush exit 2023-11-25T05:26:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.634+0000 7f959496c640 20 client.21316.objecter shutdown clearing up homeless session... 2023-11-25T05:26:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.634+0000 7f959496c640 10 client.21316.objecter successfully canceled tick 2023-11-25T05:26:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.634+0000 7f959496c640 1 -- 172.21.15.7:0/1815284713 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f957006c820 msgr2=0x7f957006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.634+0000 7f959496c640 1 --2- 172.21.15.7:0/1815284713 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f957006c820 0x7f957006ece0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9580005880 tx=0x7f9580009040 comp rx=0 tx=0).stop 2023-11-25T05:26:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.634+0000 7f959496c640 10 monclient: shutdown 2023-11-25T05:26:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.634+0000 7f959496c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:26:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.634+0000 7f959496c640 1 -- 172.21.15.7:0/1815284713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f959013f7d0 msgr2=0x7f9590132ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:26:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.634+0000 7f959496c640 1 --2- 172.21.15.7:0/1815284713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f959013f7d0 0x7f9590132ab0 secure :-1 s=READY pgs=5176 cs=0 l=1 rev1=1 crypto rx=0x7f95900b57b0 tx=0x7f9588004a40 comp rx=0 tx=0).stop 2023-11-25T05:26:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.635+0000 7f959496c640 1 -- 172.21.15.7:0/1815284713 shutdown_connections 2023-11-25T05:26:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.635+0000 7f959496c640 1 --2- 172.21.15.7:0/1815284713 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f957006c820 0x7f957006ece0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.635+0000 7f959496c640 1 --2- 172.21.15.7:0/1815284713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f959013f7d0 0x7f9590132ab0 unknown :-1 s=CLOSED pgs=5176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.635+0000 7f959496c640 1 --2- 172.21.15.7:0/1815284713 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95900b1b70 0x7f95901350d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.635+0000 7f959496c640 1 --2- 172.21.15.7:0/1815284713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9590072890 0x7f9590070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:26:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.635+0000 7f959496c640 1 -- 172.21.15.7:0/1815284713 >> 172.21.15.7:0/1815284713 conn(0x7f95900b4d20 msgr2=0x7f959013aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:26:48.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.635+0000 7f959496c640 1 -- 172.21.15.7:0/1815284713 shutdown_connections 2023-11-25T05:26:48.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.635+0000 7f959496c640 1 -- 172.21.15.7:0/1815284713 wait complete. 2023-11-25T05:26:48.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:26:48.635+0000 7f959496c640 1 librados: shutdown 2023-11-25T05:26:48.648 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1595,"stamp":"2023-11-25T05:26:47.720508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":596568,"kb_used_data":107032,"kb_used_omap":12,"kb_used_meta":489459,"kb_avail":754378152,"statfs":{"total":773094113280,"available":772483227648,"internally_reserved":0,"allocated":109600768,"data_stored":67055970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":501206579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002668"},"pg_stats":[{"pgid":"9.15","version":"0'0","reported_seq":862,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.138807+0000","last_change":"2023-11-25T05:05:46.884752+0000","last_active":"2023-11-25T05:26:16.138807+0000","last_peered":"2023-11-25T05:26:16.138807+0000","last_clean":"2023-11-25T05:26:16.138807+0000","last_became_active":"2023-11-25T05:05:46.875394+0000","last_became_peered":"2023-11-25T05:05:46.875394+0000","last_unstale":"2023-11-25T05:26:16.138807+0000","last_undegraded":"2023-11-25T05:26:16.138807+0000","last_fullsized":"2023-11-25T05:26:16.138807+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:38.240113+0000","last_change":"2023-11-25T05:26:38.240800+0000","last_active":"2023-11-25T05:26:38.240113+0000","last_peered":"2023-11-25T05:26:38.240113+0000","last_clean":"2023-11-25T05:26:38.240113+0000","last_became_active":"2023-11-25T05:05:46.923663+0000","last_became_peered":"2023-11-25T05:05:46.923663+0000","last_unstale":"2023-11-25T05:26:38.240113+0000","last_undegraded":"2023-11-25T05:26:38.240113+0000","last_fullsized":"2023-11-25T05:26:38.240113+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:38.238720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:38.238720+0000","last_clean_scrub_stamp":"2023-11-25T05:26:38.238720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:16:48.540111+0000","scrub_duration":0.031357639,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:26.249719+0000","last_change":"2023-11-25T05:26:26.250999+0000","last_active":"2023-11-25T05:26:26.249719+0000","last_peered":"2023-11-25T05:26:26.249719+0000","last_clean":"2023-11-25T05:26:26.249719+0000","last_became_active":"2023-11-25T05:05:50.968719+0000","last_became_peered":"2023-11-25T05:05:50.968719+0000","last_unstale":"2023-11-25T05:26:26.249719+0000","last_undegraded":"2023-11-25T05:26:26.249719+0000","last_fullsized":"2023-11-25T05:26:26.249719+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:26.244542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:26.244542+0000","last_clean_scrub_stamp":"2023-11-25T05:26:26.244542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:07:20.535214+0000","scrub_duration":0.16470911399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":860,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:22.444189+0000","last_change":"2023-11-25T05:26:22.444916+0000","last_active":"2023-11-25T05:26:22.444189+0000","last_peered":"2023-11-25T05:26:22.444189+0000","last_clean":"2023-11-25T05:26:22.444189+0000","last_became_active":"2023-11-25T05:06:05.084111+0000","last_became_peered":"2023-11-25T05:06:05.084111+0000","last_unstale":"2023-11-25T05:26:22.444189+0000","last_undegraded":"2023-11-25T05:26:22.444189+0000","last_fullsized":"2023-11-25T05:26:22.444189+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:22.439964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:22.439964+0000","last_clean_scrub_stamp":"2023-11-25T05:26:22.439964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:31:36.581238+0000","scrub_duration":0.053157549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":871,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:46.364433+0000","last_change":"2023-11-25T05:26:46.365167+0000","last_active":"2023-11-25T05:26:46.364433+0000","last_peered":"2023-11-25T05:26:46.364433+0000","last_clean":"2023-11-25T05:26:46.364433+0000","last_became_active":"2023-11-25T05:05:47.855679+0000","last_became_peered":"2023-11-25T05:05:47.855679+0000","last_unstale":"2023-11-25T05:26:46.364433+0000","last_undegraded":"2023-11-25T05:26:46.364433+0000","last_fullsized":"2023-11-25T05:26:46.364433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:46.363023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:46.363023+0000","last_clean_scrub_stamp":"2023-11-25T05:26:46.363023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:29:33.832022+0000","scrub_duration":0.041375328000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":886,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:34.216155+0000","last_change":"2023-11-25T05:26:34.216155+0000","last_active":"2023-11-25T05:26:34.216155+0000","last_peered":"2023-11-25T05:26:34.216155+0000","last_clean":"2023-11-25T05:26:34.216155+0000","last_became_active":"2023-11-25T05:06:09.633454+0000","last_became_peered":"2023-11-25T05:06:09.633454+0000","last_unstale":"2023-11-25T05:26:34.216155+0000","last_undegraded":"2023-11-25T05:26:34.216155+0000","last_fullsized":"2023-11-25T05:26:34.216155+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:34.214048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:34.214048+0000","last_clean_scrub_stamp":"2023-11-25T05:26:34.214048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:15:13.883268+0000","scrub_duration":0.045256073000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:23.194252+0000","last_change":"2023-11-25T05:26:23.194252+0000","last_active":"2023-11-25T05:26:23.194252+0000","last_peered":"2023-11-25T05:26:23.194252+0000","last_clean":"2023-11-25T05:26:23.194252+0000","last_became_active":"2023-11-25T05:05:53.099253+0000","last_became_peered":"2023-11-25T05:05:53.099253+0000","last_unstale":"2023-11-25T05:26:23.194252+0000","last_undegraded":"2023-11-25T05:26:23.194252+0000","last_fullsized":"2023-11-25T05:26:23.194252+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:23.192030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:23.192030+0000","last_clean_scrub_stamp":"2023-11-25T05:26:23.192030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:46:00.120705+0000","scrub_duration":0.056747789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":886,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:29.174553+0000","last_change":"2023-11-25T05:26:29.174553+0000","last_active":"2023-11-25T05:26:29.174553+0000","last_peered":"2023-11-25T05:26:29.174553+0000","last_clean":"2023-11-25T05:26:29.174553+0000","last_became_active":"2023-11-25T05:05:43.496305+0000","last_became_peered":"2023-11-25T05:05:43.496305+0000","last_unstale":"2023-11-25T05:26:29.174553+0000","last_undegraded":"2023-11-25T05:26:29.174553+0000","last_fullsized":"2023-11-25T05:26:29.174553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:29.172138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:29.172138+0000","last_clean_scrub_stamp":"2023-11-25T05:26:29.172138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:43:52.822726+0000","scrub_duration":0.038697677,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":881,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:33.893713+0000","last_change":"2023-11-25T05:26:33.894594+0000","last_active":"2023-11-25T05:26:33.893713+0000","last_peered":"2023-11-25T05:26:33.893713+0000","last_clean":"2023-11-25T05:26:33.893713+0000","last_became_active":"2023-11-25T05:05:42.672388+0000","last_became_peered":"2023-11-25T05:05:42.672388+0000","last_unstale":"2023-11-25T05:26:33.893713+0000","last_undegraded":"2023-11-25T05:26:33.893713+0000","last_fullsized":"2023-11-25T05:26:33.893713+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:33.892390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:33.892390+0000","last_clean_scrub_stamp":"2023-11-25T05:26:33.892390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:32:42.524686+0000","scrub_duration":0.077926995999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:43.208623+0000","last_change":"2023-11-25T05:26:43.209282+0000","last_active":"2023-11-25T05:26:43.208623+0000","last_peered":"2023-11-25T05:26:43.208623+0000","last_clean":"2023-11-25T05:26:43.208623+0000","last_became_active":"2023-11-25T05:05:46.959854+0000","last_became_peered":"2023-11-25T05:05:46.959854+0000","last_unstale":"2023-11-25T05:26:43.208623+0000","last_undegraded":"2023-11-25T05:26:43.208623+0000","last_fullsized":"2023-11-25T05:26:43.208623+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:43.207305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:43.207305+0000","last_clean_scrub_stamp":"2023-11-25T05:26:43.207305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:31:56.183159+0000","scrub_duration":0.031291102000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:30.893207+0000","last_change":"2023-11-25T05:26:30.893854+0000","last_active":"2023-11-25T05:26:30.893207+0000","last_peered":"2023-11-25T05:26:30.893207+0000","last_clean":"2023-11-25T05:26:30.893207+0000","last_became_active":"2023-11-25T05:05:47.753176+0000","last_became_peered":"2023-11-25T05:05:47.753176+0000","last_unstale":"2023-11-25T05:26:30.893207+0000","last_undegraded":"2023-11-25T05:26:30.893207+0000","last_fullsized":"2023-11-25T05:26:30.893207+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:30.891827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:30.891827+0000","last_clean_scrub_stamp":"2023-11-25T05:26:30.891827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:13:27.472572+0000","scrub_duration":0.076990036999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:35.832685+0000","last_change":"2023-11-25T05:26:35.833330+0000","last_active":"2023-11-25T05:26:35.832685+0000","last_peered":"2023-11-25T05:26:35.832685+0000","last_clean":"2023-11-25T05:26:35.832685+0000","last_became_active":"2023-11-25T05:05:45.597733+0000","last_became_peered":"2023-11-25T05:05:45.597733+0000","last_unstale":"2023-11-25T05:26:35.832685+0000","last_undegraded":"2023-11-25T05:26:35.832685+0000","last_fullsized":"2023-11-25T05:26:35.832685+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:35.831376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:35.831376+0000","last_clean_scrub_stamp":"2023-11-25T05:26:35.831376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T13:25:11.746685+0000","scrub_duration":0.061482579000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":876,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:33.098498+0000","last_change":"2023-11-25T05:26:33.099152+0000","last_active":"2023-11-25T05:26:33.098498+0000","last_peered":"2023-11-25T05:26:33.098498+0000","last_clean":"2023-11-25T05:26:33.098498+0000","last_became_active":"2023-11-25T05:05:47.792727+0000","last_became_peered":"2023-11-25T05:05:47.792727+0000","last_unstale":"2023-11-25T05:26:33.098498+0000","last_undegraded":"2023-11-25T05:26:33.098498+0000","last_fullsized":"2023-11-25T05:26:33.098498+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:33.097199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:33.097199+0000","last_clean_scrub_stamp":"2023-11-25T05:26:33.097199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:10:01.913154+0000","scrub_duration":0.034383390999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:44.447839+0000","last_change":"2023-11-25T05:26:44.448585+0000","last_active":"2023-11-25T05:26:44.447839+0000","last_peered":"2023-11-25T05:26:44.447839+0000","last_clean":"2023-11-25T05:26:44.447839+0000","last_became_active":"2023-11-25T05:04:33.577510+0000","last_became_peered":"2023-11-25T05:04:33.577510+0000","last_unstale":"2023-11-25T05:26:44.447839+0000","last_undegraded":"2023-11-25T05:26:44.447839+0000","last_fullsized":"2023-11-25T05:26:44.447839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:44.446380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:44.446380+0000","last_clean_scrub_stamp":"2023-11-25T05:26:44.446380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:50:58.487234+0000","scrub_duration":0.039103001999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":863,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:16.137153+0000","last_change":"2023-11-25T05:05:58.700776+0000","last_active":"2023-11-25T05:26:16.137153+0000","last_peered":"2023-11-25T05:26:16.137153+0000","last_clean":"2023-11-25T05:26:16.137153+0000","last_became_active":"2023-11-25T05:05:58.695110+0000","last_became_peered":"2023-11-25T05:05:58.695110+0000","last_unstale":"2023-11-25T05:26:16.137153+0000","last_undegraded":"2023-11-25T05:26:16.137153+0000","last_fullsized":"2023-11-25T05:26:16.137153+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:05:36.467803+0000","last_clean_scrub_stamp":"2023-11-25T05:05:36.467803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":886,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:26.960188+0000","last_change":"2023-11-25T05:26:26.961299+0000","last_active":"2023-11-25T05:26:26.960188+0000","last_peered":"2023-11-25T05:26:26.960188+0000","last_clean":"2023-11-25T05:26:26.960188+0000","last_became_active":"2023-11-25T05:04:33.610776+0000","last_became_peered":"2023-11-25T05:04:33.610776+0000","last_unstale":"2023-11-25T05:26:26.960188+0000","last_undegraded":"2023-11-25T05:26:26.960188+0000","last_fullsized":"2023-11-25T05:26:26.960188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:26.958130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:26.958130+0000","last_clean_scrub_stamp":"2023-11-25T05:26:26.958130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:07:22.646353+0000","scrub_duration":0.053590394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:29.092414+0000","last_change":"2023-11-25T05:26:29.093149+0000","last_active":"2023-11-25T05:26:29.092414+0000","last_peered":"2023-11-25T05:26:29.092414+0000","last_clean":"2023-11-25T05:26:29.092414+0000","last_became_active":"2023-11-25T05:06:12.543810+0000","last_became_peered":"2023-11-25T05:06:12.543810+0000","last_unstale":"2023-11-25T05:26:29.092414+0000","last_undegraded":"2023-11-25T05:26:29.092414+0000","last_fullsized":"2023-11-25T05:26:29.092414+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:29.091057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:29.091057+0000","last_clean_scrub_stamp":"2023-11-25T05:26:29.091057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:56:22.665568+0000","scrub_duration":0.035620111000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":876,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:40.457222+0000","last_change":"2023-11-25T05:26:40.457946+0000","last_active":"2023-11-25T05:26:40.457222+0000","last_peered":"2023-11-25T05:26:40.457222+0000","last_clean":"2023-11-25T05:26:40.457222+0000","last_became_active":"2023-11-25T05:04:33.337158+0000","last_became_peered":"2023-11-25T05:04:33.337158+0000","last_unstale":"2023-11-25T05:26:40.457222+0000","last_undegraded":"2023-11-25T05:26:40.457222+0000","last_fullsized":"2023-11-25T05:26:40.457222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:40.455774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:40.455774+0000","last_clean_scrub_stamp":"2023-11-25T05:26:40.455774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:48:38.148988+0000","scrub_duration":0.032964660999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:27.153493+0000","last_change":"2023-11-25T05:26:27.154234+0000","last_active":"2023-11-25T05:26:27.153493+0000","last_peered":"2023-11-25T05:26:27.153493+0000","last_clean":"2023-11-25T05:26:27.153493+0000","last_became_active":"2023-11-25T05:05:51.097150+0000","last_became_peered":"2023-11-25T05:05:51.097150+0000","last_unstale":"2023-11-25T05:26:27.153493+0000","last_undegraded":"2023-11-25T05:26:27.153493+0000","last_fullsized":"2023-11-25T05:26:27.153493+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:27.152003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:27.152003+0000","last_clean_scrub_stamp":"2023-11-25T05:26:27.152003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T13:29:44.174699+0000","scrub_duration":0.074299855999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":885,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:26.311099+0000","last_change":"2023-11-25T05:26:26.315627+0000","last_active":"2023-11-25T05:26:26.311099+0000","last_peered":"2023-11-25T05:26:26.311099+0000","last_clean":"2023-11-25T05:26:26.311099+0000","last_became_active":"2023-11-25T05:04:33.388849+0000","last_became_peered":"2023-11-25T05:04:33.388849+0000","last_unstale":"2023-11-25T05:26:26.311099+0000","last_undegraded":"2023-11-25T05:26:26.311099+0000","last_fullsized":"2023-11-25T05:26:26.311099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:26.307049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:26.307049+0000","last_clean_scrub_stamp":"2023-11-25T05:26:26.307049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:30:19.433152+0000","scrub_duration":0.21619841400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:31.193834+0000","last_change":"2023-11-25T05:26:31.194481+0000","last_active":"2023-11-25T05:26:31.193834+0000","last_peered":"2023-11-25T05:26:31.193834+0000","last_clean":"2023-11-25T05:26:31.193834+0000","last_became_active":"2023-11-25T05:05:55.038711+0000","last_became_peered":"2023-11-25T05:05:55.038711+0000","last_unstale":"2023-11-25T05:26:31.193834+0000","last_undegraded":"2023-11-25T05:26:31.193834+0000","last_fullsized":"2023-11-25T05:26:31.193834+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:31.192518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:31.192518+0000","last_clean_scrub_stamp":"2023-11-25T05:26:31.192518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:23:23.245247+0000","scrub_duration":0.042831777000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":876,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:29.343188+0000","last_change":"2023-11-25T05:26:29.343892+0000","last_active":"2023-11-25T05:26:29.343188+0000","last_peered":"2023-11-25T05:26:29.343188+0000","last_clean":"2023-11-25T05:26:29.343188+0000","last_became_active":"2023-11-25T05:04:33.376375+0000","last_became_peered":"2023-11-25T05:04:33.376375+0000","last_unstale":"2023-11-25T05:26:29.343188+0000","last_undegraded":"2023-11-25T05:26:29.343188+0000","last_fullsized":"2023-11-25T05:26:29.343188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:29.341842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:29.341842+0000","last_clean_scrub_stamp":"2023-11-25T05:26:29.341842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:20:11.366575+0000","scrub_duration":0.028595209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":887,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:30.155622+0000","last_change":"2023-11-25T05:26:30.156324+0000","last_active":"2023-11-25T05:26:30.155622+0000","last_peered":"2023-11-25T05:26:30.155622+0000","last_clean":"2023-11-25T05:26:30.155622+0000","last_became_active":"2023-11-25T05:05:44.554983+0000","last_became_peered":"2023-11-25T05:05:44.554983+0000","last_unstale":"2023-11-25T05:26:30.155622+0000","last_undegraded":"2023-11-25T05:26:30.155622+0000","last_fullsized":"2023-11-25T05:26:30.155622+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:30.154261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:30.154261+0000","last_clean_scrub_stamp":"2023-11-25T05:26:30.154261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:53:16.053380+0000","scrub_duration":0.054855872999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:28.100253+0000","last_change":"2023-11-25T05:26:28.100948+0000","last_active":"2023-11-25T05:26:28.100253+0000","last_peered":"2023-11-25T05:26:28.100253+0000","last_clean":"2023-11-25T05:26:28.100253+0000","last_became_active":"2023-11-25T05:04:33.196533+0000","last_became_peered":"2023-11-25T05:04:33.196533+0000","last_unstale":"2023-11-25T05:26:28.100253+0000","last_undegraded":"2023-11-25T05:26:28.100253+0000","last_fullsized":"2023-11-25T05:26:28.100253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:28.098831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:28.098831+0000","last_clean_scrub_stamp":"2023-11-25T05:26:28.098831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T13:26:33.808201+0000","scrub_duration":0.031799603000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:36.233927+0000","last_change":"2023-11-25T05:26:36.234604+0000","last_active":"2023-11-25T05:26:36.233927+0000","last_peered":"2023-11-25T05:26:36.233927+0000","last_clean":"2023-11-25T05:26:36.233927+0000","last_became_active":"2023-11-25T05:05:45.759824+0000","last_became_peered":"2023-11-25T05:05:45.759824+0000","last_unstale":"2023-11-25T05:26:36.233927+0000","last_undegraded":"2023-11-25T05:26:36.233927+0000","last_fullsized":"2023-11-25T05:26:36.233927+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:36.232550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:36.232550+0000","last_clean_scrub_stamp":"2023-11-25T05:26:36.232550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T09:59:53.238852+0000","scrub_duration":0.081003382999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":877,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:27.343406+0000","last_change":"2023-11-25T05:26:27.343406+0000","last_active":"2023-11-25T05:26:27.343406+0000","last_peered":"2023-11-25T05:26:27.343406+0000","last_clean":"2023-11-25T05:26:27.343406+0000","last_became_active":"2023-11-25T05:04:33.242068+0000","last_became_peered":"2023-11-25T05:04:33.242068+0000","last_unstale":"2023-11-25T05:26:27.343406+0000","last_undegraded":"2023-11-25T05:26:27.343406+0000","last_fullsized":"2023-11-25T05:26:27.343406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:27.341037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:27.341037+0000","last_clean_scrub_stamp":"2023-11-25T05:26:27.341037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:13:18.486294+0000","scrub_duration":0.033598854999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:27.111527+0000","last_change":"2023-11-25T05:26:27.112227+0000","last_active":"2023-11-25T05:26:27.111527+0000","last_peered":"2023-11-25T05:26:27.111527+0000","last_clean":"2023-11-25T05:26:27.111527+0000","last_became_active":"2023-11-25T05:05:45.057546+0000","last_became_peered":"2023-11-25T05:05:45.057546+0000","last_unstale":"2023-11-25T05:26:27.111527+0000","last_undegraded":"2023-11-25T05:26:27.111527+0000","last_fullsized":"2023-11-25T05:26:27.111527+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:27.110149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:27.110149+0000","last_clean_scrub_stamp":"2023-11-25T05:26:27.110149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T17:19:27.768590+0000","scrub_duration":0.035816397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":876,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:37.485974+0000","last_change":"2023-11-25T05:26:37.486693+0000","last_active":"2023-11-25T05:26:37.485974+0000","last_peered":"2023-11-25T05:26:37.485974+0000","last_clean":"2023-11-25T05:26:37.485974+0000","last_became_active":"2023-11-25T05:04:33.778042+0000","last_became_peered":"2023-11-25T05:04:33.778042+0000","last_unstale":"2023-11-25T05:26:37.485974+0000","last_undegraded":"2023-11-25T05:26:37.485974+0000","last_fullsized":"2023-11-25T05:26:37.485974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:37.484485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:37.484485+0000","last_clean_scrub_stamp":"2023-11-25T05:26:37.484485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:52:43.582599+0000","scrub_duration":0.033599663000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"918'78","reported_seq":988,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:22.189192+0000","last_change":"2023-11-25T05:26:22.189192+0000","last_active":"2023-11-25T05:26:22.189192+0000","last_peered":"2023-11-25T05:26:22.189192+0000","last_clean":"2023-11-25T05:26:22.189192+0000","last_became_active":"2023-11-25T05:04:27.388156+0000","last_became_peered":"2023-11-25T05:04:27.388156+0000","last_unstale":"2023-11-25T05:26:22.189192+0000","last_undegraded":"2023-11-25T05:26:22.189192+0000","last_fullsized":"2023-11-25T05:26:22.189192+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'78","last_scrub_stamp":"2023-11-25T05:26:22.186221+0000","last_deep_scrub":"918'78","last_deep_scrub_stamp":"2023-11-25T05:26:22.186221+0000","last_clean_scrub_stamp":"2023-11-25T05:26:22.186221+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:49:24.005907+0000","scrub_duration":0.15619770899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":883,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:27.854139+0000","last_change":"2023-11-25T05:26:27.854139+0000","last_active":"2023-11-25T05:26:27.854139+0000","last_peered":"2023-11-25T05:26:27.854139+0000","last_clean":"2023-11-25T05:26:27.854139+0000","last_became_active":"2023-11-25T05:05:47.973359+0000","last_became_peered":"2023-11-25T05:05:47.973359+0000","last_unstale":"2023-11-25T05:26:27.854139+0000","last_undegraded":"2023-11-25T05:26:27.854139+0000","last_fullsized":"2023-11-25T05:26:27.854139+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:27.849775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:27.849775+0000","last_clean_scrub_stamp":"2023-11-25T05:26:27.849775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:25:29.815659+0000","scrub_duration":0.026694712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":863,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:21.466727+0000","last_change":"2023-11-25T05:26:21.466727+0000","last_active":"2023-11-25T05:26:21.466727+0000","last_peered":"2023-11-25T05:26:21.466727+0000","last_clean":"2023-11-25T05:26:21.466727+0000","last_became_active":"2023-11-25T05:05:54.045707+0000","last_became_peered":"2023-11-25T05:05:54.045707+0000","last_unstale":"2023-11-25T05:26:21.466727+0000","last_undegraded":"2023-11-25T05:26:21.466727+0000","last_fullsized":"2023-11-25T05:26:21.466727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:21.462123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:21.462123+0000","last_clean_scrub_stamp":"2023-11-25T05:26:21.462123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:08:12.548363+0000","scrub_duration":0.061865305000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":881,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:28.145820+0000","last_change":"2023-11-25T05:26:28.146469+0000","last_active":"2023-11-25T05:26:28.145820+0000","last_peered":"2023-11-25T05:26:28.145820+0000","last_clean":"2023-11-25T05:26:28.145820+0000","last_became_active":"2023-11-25T05:05:42.487757+0000","last_became_peered":"2023-11-25T05:05:42.487757+0000","last_unstale":"2023-11-25T05:26:28.145820+0000","last_undegraded":"2023-11-25T05:26:28.145820+0000","last_fullsized":"2023-11-25T05:26:28.145820+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:28.144553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:28.144553+0000","last_clean_scrub_stamp":"2023-11-25T05:26:28.144553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T09:43:48.401548+0000","scrub_duration":0.033698147999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"918'8","reported_seq":892,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:31.158768+0000","last_change":"2023-11-25T05:26:31.159600+0000","last_active":"2023-11-25T05:26:31.158768+0000","last_peered":"2023-11-25T05:26:31.158768+0000","last_clean":"2023-11-25T05:26:31.158768+0000","last_became_active":"2023-11-25T05:05:45.212870+0000","last_became_peered":"2023-11-25T05:05:45.212870+0000","last_unstale":"2023-11-25T05:26:31.158768+0000","last_undegraded":"2023-11-25T05:26:31.158768+0000","last_fullsized":"2023-11-25T05:26:31.158768+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'8","last_scrub_stamp":"2023-11-25T05:26:31.157416+0000","last_deep_scrub":"918'8","last_deep_scrub_stamp":"2023-11-25T05:26:31.157416+0000","last_clean_scrub_stamp":"2023-11-25T05:26:31.157416+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:20:18.589017+0000","scrub_duration":0.072406576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:24.156130+0000","last_change":"2023-11-25T05:26:24.156858+0000","last_active":"2023-11-25T05:26:24.156130+0000","last_peered":"2023-11-25T05:26:24.156130+0000","last_clean":"2023-11-25T05:26:24.156130+0000","last_became_active":"2023-11-25T05:05:49.941593+0000","last_became_peered":"2023-11-25T05:05:49.941593+0000","last_unstale":"2023-11-25T05:26:24.156130+0000","last_undegraded":"2023-11-25T05:26:24.156130+0000","last_fullsized":"2023-11-25T05:26:24.156130+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:24.154721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:24.154721+0000","last_clean_scrub_stamp":"2023-11-25T05:26:24.154721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:11:07.189026+0000","scrub_duration":0.037000584000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:30.074375+0000","last_change":"2023-11-25T05:26:30.075111+0000","last_active":"2023-11-25T05:26:30.074375+0000","last_peered":"2023-11-25T05:26:30.074375+0000","last_clean":"2023-11-25T05:26:30.074375+0000","last_became_active":"2023-11-25T05:06:26.500023+0000","last_became_peered":"2023-11-25T05:06:26.500023+0000","last_unstale":"2023-11-25T05:26:30.074375+0000","last_undegraded":"2023-11-25T05:26:30.074375+0000","last_fullsized":"2023-11-25T05:26:30.074375+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:30.072946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:30.072946+0000","last_clean_scrub_stamp":"2023-11-25T05:26:30.072946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T09:35:15.693042+0000","scrub_duration":0.060247930999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":877,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:30.118560+0000","last_change":"2023-11-25T05:26:30.118560+0000","last_active":"2023-11-25T05:26:30.118560+0000","last_peered":"2023-11-25T05:26:30.118560+0000","last_clean":"2023-11-25T05:26:30.118560+0000","last_became_active":"2023-11-25T05:05:48.093083+0000","last_became_peered":"2023-11-25T05:05:48.093083+0000","last_unstale":"2023-11-25T05:26:30.118560+0000","last_undegraded":"2023-11-25T05:26:30.118560+0000","last_fullsized":"2023-11-25T05:26:30.118560+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:30.116574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:30.116574+0000","last_clean_scrub_stamp":"2023-11-25T05:26:30.116574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:57:04.485129+0000","scrub_duration":0.033886922999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:35.168400+0000","last_change":"2023-11-25T05:26:35.168400+0000","last_active":"2023-11-25T05:26:35.168400+0000","last_peered":"2023-11-25T05:26:35.168400+0000","last_clean":"2023-11-25T05:26:35.168400+0000","last_became_active":"2023-11-25T05:05:55.006070+0000","last_became_peered":"2023-11-25T05:05:55.006070+0000","last_unstale":"2023-11-25T05:26:35.168400+0000","last_undegraded":"2023-11-25T05:26:35.168400+0000","last_fullsized":"2023-11-25T05:26:35.168400+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:35.166401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:35.166401+0000","last_clean_scrub_stamp":"2023-11-25T05:26:35.166401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:53:47.955364+0000","scrub_duration":0.045775001000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:34.143819+0000","last_change":"2023-11-25T05:26:34.144478+0000","last_active":"2023-11-25T05:26:34.143819+0000","last_peered":"2023-11-25T05:26:34.143819+0000","last_clean":"2023-11-25T05:26:34.143819+0000","last_became_active":"2023-11-25T05:05:42.327418+0000","last_became_peered":"2023-11-25T05:05:42.327418+0000","last_unstale":"2023-11-25T05:26:34.143819+0000","last_undegraded":"2023-11-25T05:26:34.143819+0000","last_fullsized":"2023-11-25T05:26:34.143819+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:34.138526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:34.138526+0000","last_clean_scrub_stamp":"2023-11-25T05:26:34.138526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:13:40.861506+0000","scrub_duration":0.030268010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:31.961952+0000","last_change":"2023-11-25T05:26:31.962675+0000","last_active":"2023-11-25T05:26:31.961952+0000","last_peered":"2023-11-25T05:26:31.961952+0000","last_clean":"2023-11-25T05:26:31.961952+0000","last_became_active":"2023-11-25T05:05:45.334024+0000","last_became_peered":"2023-11-25T05:05:45.334024+0000","last_unstale":"2023-11-25T05:26:31.961952+0000","last_undegraded":"2023-11-25T05:26:31.961952+0000","last_fullsized":"2023-11-25T05:26:31.961952+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:31.959134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:31.959134+0000","last_clean_scrub_stamp":"2023-11-25T05:26:31.959134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T09:57:27.882194+0000","scrub_duration":0.074298776999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:32.028935+0000","last_change":"2023-11-25T05:26:32.030103+0000","last_active":"2023-11-25T05:26:32.028935+0000","last_peered":"2023-11-25T05:26:32.028935+0000","last_clean":"2023-11-25T05:26:32.028935+0000","last_became_active":"2023-11-25T05:06:27.122861+0000","last_became_peered":"2023-11-25T05:06:27.122861+0000","last_unstale":"2023-11-25T05:26:32.028935+0000","last_undegraded":"2023-11-25T05:26:32.028935+0000","last_fullsized":"2023-11-25T05:26:32.028935+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:32.026758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:32.026758+0000","last_clean_scrub_stamp":"2023-11-25T05:26:32.026758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:34:00.052329+0000","scrub_duration":0.043541608000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":881,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:36.853733+0000","last_change":"2023-11-25T05:26:36.854435+0000","last_active":"2023-11-25T05:26:36.853733+0000","last_peered":"2023-11-25T05:26:36.853733+0000","last_clean":"2023-11-25T05:26:36.853733+0000","last_became_active":"2023-11-25T05:05:57.177615+0000","last_became_peered":"2023-11-25T05:05:57.177615+0000","last_unstale":"2023-11-25T05:26:36.853733+0000","last_undegraded":"2023-11-25T05:26:36.853733+0000","last_fullsized":"2023-11-25T05:26:36.853733+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:36.852342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:36.852342+0000","last_clean_scrub_stamp":"2023-11-25T05:26:36.852342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T13:33:40.863507+0000","scrub_duration":0.050814239999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":16,"seq":68719477005,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66664,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305176,"statfs":{"total":96636764160,"available":96568500224,"internally_reserved":0,"allocated":13598720,"data_stored":8282535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":17.164000000000001,"5min":25.925000000000001,"15min":25.350999999999999},"min":{"1min":4.9020000000000001,"5min":4.9020000000000001,"15min":4.9020000000000001},"max":{"1min":100.782,"5min":164.57300000000001,"15min":203.28800000000001},"last":8.1799999999999997},{"interface":"front","average":{"1min":15.237,"5min":26.946000000000002,"15min":26.018000000000001},"min":{"1min":5.7000000000000002,"5min":5.6859999999999999,"15min":5.4400000000000004},"max":{"1min":92.989000000000004,"5min":238.51300000000001,"15min":347.452},"last":8.9770000000000003}]},{"osd":1,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":20.888000000000002,"5min":26.521999999999998,"15min":25.02},"min":{"1min":8.3249999999999993,"5min":7.4660000000000002,"15min":5.6589999999999998},"max":{"1min":86.962999999999994,"5min":287.27199999999999,"15min":287.27199999999999},"last":13.428000000000001},{"interface":"front","average":{"1min":17.350999999999999,"5min":27.222000000000001,"15min":23.896000000000001},"min":{"1min":5.282,"5min":5.282,"15min":5.282},"max":{"1min":57.453000000000003,"5min":286.57400000000001,"15min":286.57400000000001},"last":11.019}]},{"osd":3,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":23.404,"5min":20.637,"15min":20.605},"min":{"1min":6.3289999999999997,"5min":6.3289999999999997,"15min":5.9459999999999997},"max":{"1min":167.83799999999999,"5min":167.83799999999999,"15min":177.66},"last":17.765000000000001},{"interface":"front","average":{"1min":30.977,"5min":28.187999999999999,"15min":31.155999999999999},"min":{"1min":11.647,"5min":8.1829999999999998,"15min":7.609},"max":{"1min":156.17699999999999,"5min":156.17699999999999,"15min":273.99900000000002},"last":14.193}]},{"osd":4,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":20.108000000000001,"5min":29.747,"15min":26.818000000000001},"min":{"1min":7.7789999999999999,"5min":6.9109999999999996,"15min":6.9109999999999996},"max":{"1min":71.611000000000004,"5min":275.66300000000001,"15min":349.90300000000002},"last":11.397},{"interface":"front","average":{"1min":17.427,"5min":26.164999999999999,"15min":22.431000000000001},"min":{"1min":7.1589999999999998,"5min":7.1589999999999998,"15min":6.6760000000000002},"max":{"1min":73.219999999999999,"5min":285.12700000000001,"15min":285.12700000000001},"last":25.039000000000001}]},{"osd":5,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":22.722999999999999,"5min":24.5,"15min":29.382999999999999},"min":{"1min":11.710000000000001,"5min":9.7840000000000007,"15min":8.0619999999999994},"max":{"1min":89.492000000000004,"5min":138.34899999999999,"15min":353.63400000000001},"last":14.879},{"interface":"front","average":{"1min":26.366,"5min":30.738,"15min":34.677999999999997},"min":{"1min":14.381,"5min":13.541,"15min":11.539999999999999},"max":{"1min":96.620999999999995,"5min":146.41300000000001,"15min":354.56200000000001},"last":18.564}]},{"osd":6,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":31.605,"5min":30.536999999999999,"15min":31.172000000000001},"min":{"1min":16.321000000000002,"5min":11.292999999999999,"15min":9.7590000000000003},"max":{"1min":153.01400000000001,"5min":153.01400000000001,"15min":213.11000000000001},"last":16.872},{"interface":"front","average":{"1min":33.220999999999997,"5min":34.654000000000003,"15min":34.892000000000003},"min":{"1min":15.217000000000001,"5min":12.462999999999999,"15min":12.462999999999999},"max":{"1min":144.51400000000001,"5min":288.30000000000001,"15min":288.30000000000001},"last":19.818999999999999}]},{"osd":7,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":25.690999999999999,"5min":27.300000000000001,"15min":26.981000000000002},"min":{"1min":11.557,"5min":6.375,"15min":6.375},"max":{"1min":90.355999999999995,"5min":216.095,"15min":350.81599999999997},"last":12.244},{"interface":"front","average":{"1min":21.873000000000001,"5min":22.734000000000002,"15min":23.170000000000002},"min":{"1min":8.6400000000000006,"5min":7.5419999999999998,"15min":7.1669999999999998},"max":{"1min":96.100999999999999,"5min":141.59100000000001,"15min":210.55500000000001},"last":13.038}]}]},{"osd":1,"up_from":16,"seq":68719477005,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67056,"kb_used_data":13672,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94304784,"statfs":{"total":96636764160,"available":96568098816,"internally_reserved":0,"allocated":14000128,"data_stored":8680375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":23.388000000000002,"5min":20.16,"15min":21.632000000000001},"min":{"1min":6.165,"5min":5.9809999999999999,"15min":5.1600000000000001},"max":{"1min":163.62,"5min":221.589,"15min":349.75799999999998},"last":15.331},{"interface":"front","average":{"1min":32.435000000000002,"5min":29.16,"15min":26.466999999999999},"min":{"1min":9.4299999999999997,"5min":7.1529999999999996,"15min":6.5309999999999997},"max":{"1min":240.31200000000001,"5min":318.02600000000001,"15min":318.02600000000001},"last":17.481999999999999}]},{"osd":2,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":28.914000000000001,"5min":20.263000000000002,"15min":22.899999999999999},"min":{"1min":6.7889999999999997,"5min":5.1429999999999998,"15min":4.9619999999999997},"max":{"1min":148.28299999999999,"5min":161.10900000000001,"15min":353.286},"last":10.544},{"interface":"front","average":{"1min":34.442,"5min":21.100999999999999,"15min":19.414999999999999},"min":{"1min":7.7750000000000004,"5min":6.3280000000000003,"15min":5.7149999999999999},"max":{"1min":169.84100000000001,"5min":169.84100000000001,"15min":247.99199999999999},"last":11.311999999999999}]},{"osd":3,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":30.649000000000001,"5min":28.231999999999999,"15min":24.68},"min":{"1min":7.0339999999999998,"5min":6.665,"15min":6.5899999999999999},"max":{"1min":169.172,"5min":244.58199999999999,"15min":440.05599999999998},"last":7.8810000000000002},{"interface":"front","average":{"1min":31.716000000000001,"5min":30.024000000000001,"15min":28.202999999999999},"min":{"1min":8.4369999999999994,"5min":8.4369999999999994,"15min":5.9889999999999999},"max":{"1min":168.58000000000001,"5min":242.29400000000001,"15min":383.39800000000002},"last":16.568999999999999}]},{"osd":4,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":20.422000000000001,"5min":17.603000000000002,"15min":21.190999999999999},"min":{"1min":8.9909999999999997,"5min":6.7830000000000004,"15min":6.5789999999999997},"max":{"1min":63.802999999999997,"5min":113.84,"15min":360.161},"last":8.8960000000000008},{"interface":"front","average":{"1min":27.309999999999999,"5min":24.93,"15min":23.541},"min":{"1min":9.0510000000000002,"5min":9.0510000000000002,"15min":7.4409999999999998},"max":{"1min":114.33199999999999,"5min":226.68000000000001,"15min":364.39800000000002},"last":19.245999999999999}]},{"osd":5,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":39.484999999999999,"5min":28.780000000000001,"15min":29.396999999999998},"min":{"1min":9.6669999999999998,"5min":9.6669999999999998,"15min":7.4359999999999999},"max":{"1min":157.357,"5min":157.357,"15min":282.63900000000001},"last":20.234999999999999},{"interface":"front","average":{"1min":29.167999999999999,"5min":24.120999999999999,"15min":26.896000000000001},"min":{"1min":7.6630000000000003,"5min":7.2889999999999997,"15min":7.1529999999999996},"max":{"1min":105.211,"5min":163.33600000000001,"15min":382.01400000000001},"last":12.101000000000001}]},{"osd":6,"last update":"Sat Nov 25 05:26:23 2023","interfaces":[{"interface":"back","average":{"1min":22.631,"5min":23.855,"15min":27.713999999999999},"min":{"1min":9.2360000000000007,"5min":7.5209999999999999,"15min":6.8230000000000004},"max":{"1min":71.406999999999996,"5min":187.62100000000001,"15min":493.18799999999999},"last":13.898999999999999},{"interface":"front","average":{"1min":26.407,"5min":26.518999999999998,"15min":30.838000000000001},"min":{"1min":13.795,"5min":9.9260000000000002,"15min":9.423},"max":{"1min":75.549999999999997,"5min":131.99600000000001,"15min":387.71199999999999},"last":14.535}]},{"osd":7,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":23.481999999999999,"5min":23.608000000000001,"15min":24.239999999999998},"min":{"1min":8.8559999999999999,"5min":7.0279999999999996,"15min":7.0279999999999996},"max":{"1min":116.39400000000001,"5min":219.57900000000001,"15min":272.03699999999998},"last":9.8729999999999993},{"interface":"front","average":{"1min":30.045999999999999,"5min":26.843,"15min":27.969000000000001},"min":{"1min":11.65,"5min":10.239000000000001,"15min":8.3889999999999993},"max":{"1min":118.447,"5min":126.80200000000001,"15min":297.976},"last":18.366}]}]},{"osd":3,"up_from":16,"seq":68719477006,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84712,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94287128,"statfs":{"total":96636764160,"available":96550019072,"internally_reserved":0,"allocated":13598720,"data_stored":8282535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":21.867000000000001,"5min":16.334,"15min":21.956},"min":{"1min":5.556,"5min":4.3090000000000002,"15min":4.3090000000000002},"max":{"1min":100.45399999999999,"5min":124.46299999999999,"15min":394.84199999999998},"last":4.5279999999999996},{"interface":"front","average":{"1min":19.355,"5min":17.161000000000001,"15min":21.390999999999998},"min":{"1min":7.2969999999999997,"5min":4.9020000000000001,"15min":4.5890000000000004},"max":{"1min":84.197000000000003,"5min":119.611,"15min":379.51900000000001},"last":10.741}]},{"osd":1,"last update":"Sat Nov 25 05:26:14 2023","interfaces":[{"interface":"back","average":{"1min":21.181999999999999,"5min":20.585000000000001,"15min":28.129999999999999},"min":{"1min":7.6379999999999999,"5min":7.1779999999999999,"15min":5.4080000000000004},"max":{"1min":75.337999999999994,"5min":132.477,"15min":674.39599999999996},"last":19.164000000000001},{"interface":"front","average":{"1min":18.774000000000001,"5min":24.016999999999999,"15min":26.608000000000001},"min":{"1min":6.1829999999999998,"5min":6.1829999999999998,"15min":6.1829999999999998},"max":{"1min":62.155999999999999,"5min":253.03100000000001,"15min":399.98500000000001},"last":11.407}]},{"osd":2,"last update":"Sat Nov 25 05:26:23 2023","interfaces":[{"interface":"back","average":{"1min":25.303000000000001,"5min":17.934000000000001,"15min":29.198},"min":{"1min":8.8580000000000005,"5min":7.2649999999999997,"15min":7.0620000000000003},"max":{"1min":84.911000000000001,"5min":84.911000000000001,"15min":673.12400000000002},"last":9.3170000000000002},{"interface":"front","average":{"1min":26.186,"5min":23.507000000000001,"15min":29.689},"min":{"1min":9.1300000000000008,"5min":7.351,"15min":7.351},"max":{"1min":79.689999999999998,"5min":227.36199999999999,"15min":462.27699999999999},"last":16.215}]},{"osd":4,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":21.428999999999998,"5min":21.751999999999999,"15min":31.024000000000001},"min":{"1min":8.8360000000000003,"5min":8.8360000000000003,"15min":7.6890000000000001},"max":{"1min":81.578000000000003,"5min":141.38399999999999,"15min":676.21299999999997},"last":17.466000000000001},{"interface":"front","average":{"1min":19.236000000000001,"5min":20.013999999999999,"15min":23.495999999999999},"min":{"1min":7.6040000000000001,"5min":6.641,"15min":6.1559999999999997},"max":{"1min":77.034999999999997,"5min":313.18299999999999,"15min":313.18299999999999},"last":8.0999999999999996}]},{"osd":5,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":26.818000000000001,"5min":23.085999999999999,"15min":32.587000000000003},"min":{"1min":10.710000000000001,"5min":9.6999999999999993,"15min":9.3360000000000003},"max":{"1min":80.661000000000001,"5min":194.51400000000001,"15min":675.37800000000004},"last":20.728000000000002},{"interface":"front","average":{"1min":27.657,"5min":21.199999999999999,"15min":28.454999999999998},"min":{"1min":11.975,"5min":7.5190000000000001,"15min":7.5190000000000001},"max":{"1min":103.021,"5min":131.768,"15min":314.40800000000002},"last":18.303999999999998}]},{"osd":6,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":23.952999999999999,"5min":20.739999999999998,"15min":24.882000000000001},"min":{"1min":8.5519999999999996,"5min":8.3030000000000008,"15min":7.0190000000000001},"max":{"1min":83.287999999999997,"5min":172.66,"15min":312.20699999999999},"last":15.236000000000001},{"interface":"front","average":{"1min":26.363,"5min":24.664000000000001,"15min":35.289999999999999},"min":{"1min":11.327,"5min":10.263999999999999,"15min":10.029},"max":{"1min":102.366,"5min":185.809,"15min":677.03499999999997},"last":10.135}]},{"osd":7,"last update":"Sat Nov 25 05:26:05 2023","interfaces":[{"interface":"back","average":{"1min":23.309000000000001,"5min":23.741,"15min":28.478999999999999},"min":{"1min":11.162000000000001,"5min":8.1609999999999996,"15min":7.6420000000000003},"max":{"1min":68.536000000000001,"5min":315.61500000000001,"15min":315.61500000000001},"last":13.803000000000001},{"interface":"front","average":{"1min":26.361999999999998,"5min":24.396000000000001,"15min":30.718},"min":{"1min":10.535,"5min":10.535,"15min":10.449999999999999},"max":{"1min":102.361,"5min":145.86099999999999,"15min":320.38400000000001},"last":12.917999999999999}]}]},{"osd":0,"up_from":16,"seq":68719477007,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66664,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305176,"statfs":{"total":96636764160,"available":96568500224,"internally_reserved":0,"allocated":13598720,"data_stored":8282535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":12.904999999999999,"5min":15.397,"15min":17.724},"min":{"1min":4.6100000000000003,"5min":4.6100000000000003,"15min":4.6100000000000003},"max":{"1min":39.119,"5min":109.95399999999999,"15min":209.29900000000001},"last":7.6719999999999997},{"interface":"front","average":{"1min":12.183999999999999,"5min":16.882000000000001,"15min":19.984999999999999},"min":{"1min":5.6219999999999999,"5min":4.5869999999999997,"15min":4.4420000000000002},"max":{"1min":56.362000000000002,"5min":119.73,"15min":401.36000000000001},"last":16.373000000000001}]},{"osd":2,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":17.683,"5min":19.295000000000002,"15min":22.402999999999999},"min":{"1min":6.6680000000000001,"5min":6.5380000000000003,"15min":6.1260000000000003},"max":{"1min":71.555999999999997,"5min":94.311999999999998,"15min":460.572},"last":9.7989999999999995},{"interface":"front","average":{"1min":17.940000000000001,"5min":18.923999999999999,"15min":22.27},"min":{"1min":7.4980000000000002,"5min":6.5279999999999996,"15min":6.3090000000000002},"max":{"1min":73.290000000000006,"5min":89.322999999999993,"15min":405.94099999999997},"last":9.1590000000000007}]},{"osd":3,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":21.096,"5min":23.585999999999999,"15min":24.260000000000002},"min":{"1min":7.2999999999999998,"5min":7.2999999999999998,"15min":6.8540000000000001},"max":{"1min":53.616999999999997,"5min":157.65299999999999,"15min":267.74599999999998},"last":10.436},{"interface":"front","average":{"1min":19.989000000000001,"5min":23.132000000000001,"15min":23.312000000000001},"min":{"1min":9.8480000000000008,"5min":7.1269999999999998,"15min":6.4080000000000004},"max":{"1min":55.122,"5min":128.14500000000001,"15min":266.81900000000002},"last":17.027999999999999}]},{"osd":4,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":21.091000000000001,"5min":20.103999999999999,"15min":20.606000000000002},"min":{"1min":9.4649999999999999,"5min":8.3940000000000001,"15min":6.2750000000000004},"max":{"1min":72.540999999999997,"5min":115.369,"15min":147.86099999999999},"last":8.5459999999999994},{"interface":"front","average":{"1min":26.452000000000002,"5min":23.149999999999999,"15min":22.498999999999999},"min":{"1min":9.2189999999999994,"5min":6.8250000000000002,"15min":6.8250000000000002},"max":{"1min":110.93300000000001,"5min":127.291,"15min":148.96199999999999},"last":13.116}]},{"osd":5,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.902999999999999,"5min":23.966999999999999,"15min":23.213999999999999},"min":{"1min":9.0109999999999992,"5min":6.9829999999999997,"15min":6.9829999999999997},"max":{"1min":69.957999999999998,"5min":184.80000000000001,"15min":184.80000000000001},"last":11.048999999999999},{"interface":"front","average":{"1min":20.024000000000001,"5min":23.408999999999999,"15min":25.248000000000001},"min":{"1min":10.836,"5min":7.3849999999999998,"15min":7.3849999999999998},"max":{"1min":58.210000000000001,"5min":131.583,"15min":395.65100000000001},"last":18.065999999999999}]},{"osd":6,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":21.329000000000001,"5min":23.550000000000001,"15min":24.617000000000001},"min":{"1min":9.2829999999999995,"5min":8.4809999999999999,"15min":8.4809999999999999},"max":{"1min":71.225999999999999,"5min":116.63800000000001,"15min":161.24299999999999},"last":11.651999999999999},{"interface":"front","average":{"1min":26.015999999999998,"5min":25.968,"15min":26.870000000000001},"min":{"1min":13.154999999999999,"5min":9.7759999999999998,"15min":9.7759999999999998},"max":{"1min":71.831999999999994,"5min":115.986,"15min":138.46799999999999},"last":12.355}]},{"osd":7,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.158000000000001,"5min":18.911000000000001,"15min":20.722999999999999},"min":{"1min":10.941000000000001,"5min":8.1340000000000003,"15min":7.1079999999999997},"max":{"1min":40.393999999999998,"5min":112.73099999999999,"15min":140.595},"last":13.898},{"interface":"front","average":{"1min":21.975999999999999,"5min":22.829999999999998,"15min":25.202999999999999},"min":{"1min":11.441000000000001,"5min":10.888999999999999,"15min":9.9860000000000007},"max":{"1min":57.088000000000001,"5min":103.19,"15min":339.298},"last":18.722999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509710,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84712,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94287128,"statfs":{"total":96636764160,"available":96550019072,"internally_reserved":0,"allocated":13598720,"data_stored":8282535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":16.198,"5min":17.219999999999999,"15min":19.709},"min":{"1min":4.7489999999999997,"5min":4.6740000000000004,"15min":4.4409999999999998},"max":{"1min":117.05500000000001,"5min":117.05500000000001,"15min":203.857},"last":14.693},{"interface":"front","average":{"1min":15.382999999999999,"5min":16.821000000000002,"15min":20.439},"min":{"1min":5.4820000000000002,"5min":5.1210000000000004,"15min":4.6289999999999996},"max":{"1min":117.795,"5min":236.78700000000001,"15min":381.43900000000002},"last":6.431}]},{"osd":1,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.943999999999999,"5min":22.879000000000001,"15min":26.055},"min":{"1min":8.0890000000000004,"5min":6.2839999999999998,"15min":6.165},"max":{"1min":143.61099999999999,"5min":229.53399999999999,"15min":310.423},"last":12.734999999999999},{"interface":"front","average":{"1min":20.885999999999999,"5min":20.399999999999999,"15min":26.382999999999999},"min":{"1min":6.3360000000000003,"5min":6.2030000000000003,"15min":5.1310000000000002},"max":{"1min":190.029,"5min":243.74700000000001,"15min":548.03599999999994},"last":9.9499999999999993}]},{"osd":2,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":28.033999999999999,"5min":22.832000000000001,"15min":27.013999999999999},"min":{"1min":7.9550000000000001,"5min":6.9690000000000003,"15min":6.9690000000000003},"max":{"1min":122.754,"5min":251.524,"15min":353.80399999999997},"last":13.917999999999999},{"interface":"front","average":{"1min":26.027000000000001,"5min":27.611999999999998,"15min":30.841999999999999},"min":{"1min":10.65,"5min":10.65,"15min":10.617000000000001},"max":{"1min":120.961,"5min":249.97399999999999,"15min":304.49900000000002},"last":16.248000000000001}]},{"osd":3,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":25.802,"5min":31.065999999999999,"15min":32.701999999999998},"min":{"1min":11.24,"5min":10.164,"15min":7.5419999999999998},"max":{"1min":185.28299999999999,"5min":259.89299999999997,"15min":295.82799999999997},"last":15.484999999999999},{"interface":"front","average":{"1min":22.247,"5min":26.661000000000001,"15min":26.797999999999998},"min":{"1min":7.7720000000000002,"5min":6.9199999999999999,"15min":6.0209999999999999},"max":{"1min":168.56100000000001,"5min":249.07400000000001,"15min":266.40199999999999},"last":8.2249999999999996}]},{"osd":4,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":31.027999999999999,"5min":31.033000000000001,"15min":32.439999999999998},"min":{"1min":10.000999999999999,"5min":10.000999999999999,"15min":10.000999999999999},"max":{"1min":186.28,"5min":277.85000000000002,"15min":330.86900000000003},"last":17.050999999999998},{"interface":"front","average":{"1min":25.614999999999998,"5min":26.050000000000001,"15min":25.638000000000002},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":6.5919999999999996},"max":{"1min":192.25999999999999,"5min":288.13099999999997,"15min":288.13099999999997},"last":11.286}]},{"osd":5,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":20.140000000000001,"5min":18.035,"15min":22.939},"min":{"1min":6.0039999999999996,"5min":5.5590000000000002,"15min":5.5590000000000002},"max":{"1min":121.31399999999999,"5min":237.46100000000001,"15min":281.13999999999999},"last":9.2189999999999994},{"interface":"front","average":{"1min":25.876000000000001,"5min":28.905999999999999,"15min":33.758000000000003},"min":{"1min":8.0069999999999997,"5min":8.0069999999999997,"15min":8.0069999999999997},"max":{"1min":133.44999999999999,"5min":269.77100000000002,"15min":311.09199999999998},"last":17.824999999999999}]},{"osd":7,"last update":"Sat Nov 25 05:26:00 2023","interfaces":[{"interface":"back","average":{"1min":29.199999999999999,"5min":31.532,"15min":34.887999999999998},"min":{"1min":13.699,"5min":12.319000000000001,"15min":9.2210000000000001},"max":{"1min":140.25200000000001,"5min":259.87700000000001,"15min":367.24900000000002},"last":18.696999999999999},{"interface":"front","average":{"1min":25.48,"5min":24.911999999999999,"15min":28.138999999999999},"min":{"1min":8.8529999999999998,"5min":8.3469999999999995,"15min":7.7199999999999998},"max":{"1min":161.37899999999999,"5min":269.14400000000001,"15min":325.54199999999997},"last":12.018000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509712,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70828,"kb_used_data":13284,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94301012,"statfs":{"total":96636764160,"available":96564236288,"internally_reserved":0,"allocated":13602816,"data_stored":8282540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:26 2023","interfaces":[{"interface":"back","average":{"1min":18.672999999999998,"5min":18.614000000000001,"15min":19.898},"min":{"1min":4.8550000000000004,"5min":4.3170000000000002,"15min":3.8279999999999998},"max":{"1min":124.38200000000001,"5min":150.73599999999999,"15min":219.172},"last":5.774},{"interface":"front","average":{"1min":19.463999999999999,"5min":17.724,"15min":18.888000000000002},"min":{"1min":4.9299999999999997,"5min":4.5369999999999999,"15min":4.1289999999999996},"max":{"1min":204.953,"5min":216.86600000000001,"15min":216.86600000000001},"last":8.8840000000000003}]},{"osd":1,"last update":"Sat Nov 25 05:26:26 2023","interfaces":[{"interface":"back","average":{"1min":15.097,"5min":18.919,"15min":18.690000000000001},"min":{"1min":5.79,"5min":5.2149999999999999,"15min":4.9749999999999996},"max":{"1min":89.542000000000002,"5min":205.36500000000001,"15min":205.36500000000001},"last":7.8890000000000002},{"interface":"front","average":{"1min":25.373000000000001,"5min":21.013999999999999,"15min":20.709},"min":{"1min":8.6920000000000002,"5min":6.5759999999999996,"15min":6.5759999999999996},"max":{"1min":196.667,"5min":196.667,"15min":196.667},"last":15.804}]},{"osd":2,"last update":"Sat Nov 25 05:26:26 2023","interfaces":[{"interface":"back","average":{"1min":28.908000000000001,"5min":21.404,"15min":23.119},"min":{"1min":8.8520000000000003,"5min":6.9320000000000004,"15min":6.9320000000000004},"max":{"1min":277.51299999999998,"5min":277.51299999999998,"15min":277.51299999999998},"last":17.137},{"interface":"front","average":{"1min":31.375,"5min":24.786999999999999,"15min":26.934000000000001},"min":{"1min":14.608000000000001,"5min":10.664,"15min":8.5899999999999999},"max":{"1min":256.63,"5min":256.63,"15min":312.05200000000002},"last":19.678000000000001}]},{"osd":3,"last update":"Sat Nov 25 05:26:26 2023","interfaces":[{"interface":"back","average":{"1min":25.312000000000001,"5min":23.844000000000001,"15min":26.222999999999999},"min":{"1min":6.8760000000000003,"5min":6.8760000000000003,"15min":6.133},"max":{"1min":210.71299999999999,"5min":210.71299999999999,"15min":359.23700000000002},"last":11.172000000000001},{"interface":"front","average":{"1min":21.283999999999999,"5min":24.105,"15min":23.407},"min":{"1min":6.1719999999999997,"5min":6.1719999999999997,"15min":6.1719999999999997},"max":{"1min":137.21199999999999,"5min":196.55000000000001,"15min":381.53100000000001},"last":10.481999999999999}]},{"osd":5,"last update":"Sat Nov 25 05:25:56 2023","interfaces":[{"interface":"back","average":{"1min":18.43,"5min":20.623999999999999,"15min":26.562000000000001},"min":{"1min":5.875,"5min":5.875,"15min":5.875},"max":{"1min":72.236999999999995,"5min":195.851,"15min":360.387},"last":13.215},{"interface":"front","average":{"1min":24.617000000000001,"5min":24.105,"15min":29.986999999999998},"min":{"1min":12.101000000000001,"5min":9.4469999999999992,"15min":9.4469999999999992},"max":{"1min":122.352,"5min":159.67099999999999,"15min":274.38200000000001},"last":17.914000000000001}]},{"osd":6,"last update":"Sat Nov 25 05:25:56 2023","interfaces":[{"interface":"back","average":{"1min":20.521000000000001,"5min":24.015000000000001,"15min":27.704999999999998},"min":{"1min":10.093,"5min":10.093,"15min":9.8409999999999993},"max":{"1min":48.161000000000001,"5min":169.44499999999999,"15min":181.43299999999999},"last":18.757999999999999},{"interface":"front","average":{"1min":24.329000000000001,"5min":22.428999999999998,"15min":23.622},"min":{"1min":15.603999999999999,"5min":7.1619999999999999,"15min":5.6390000000000002},"max":{"1min":60.040999999999997,"5min":212.25399999999999,"15min":219.26499999999999},"last":13.948}]},{"osd":7,"last update":"Sat Nov 25 05:25:56 2023","interfaces":[{"interface":"back","average":{"1min":14.73,"5min":23.181000000000001,"15min":25.620999999999999},"min":{"1min":8.7100000000000009,"5min":7.0369999999999999,"15min":7.0369999999999999},"max":{"1min":27.728999999999999,"5min":202.363,"15min":215.512},"last":15.032},{"interface":"front","average":{"1min":14.417999999999999,"5min":20.943999999999999,"15min":24.698},"min":{"1min":6.0350000000000001,"5min":6.0350000000000001,"15min":6.0350000000000001},"max":{"1min":29.920999999999999,"5min":211.62200000000001,"15min":248.83199999999999},"last":11.862}]}]},{"osd":7,"up_from":13,"seq":55834575118,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66668,"kb_used_data":13284,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305172,"statfs":{"total":96636764160,"available":96568496128,"internally_reserved":0,"allocated":13602816,"data_stored":8282540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":11.608000000000001,"5min":13.819000000000001,"15min":17.303999999999998},"min":{"1min":4.6150000000000002,"5min":4.3579999999999997,"15min":4.3579999999999997},"max":{"1min":23.535,"5min":97.569000000000003,"15min":222.50800000000001},"last":9.0869999999999997},{"interface":"front","average":{"1min":9.8520000000000003,"5min":15.375999999999999,"15min":18.193999999999999},"min":{"1min":4.7450000000000001,"5min":4.6600000000000001,"15min":4.5579999999999998},"max":{"1min":25.248000000000001,"5min":218.03999999999999,"15min":218.03999999999999},"last":10.6}]},{"osd":1,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.643000000000001,"5min":13.715999999999999,"15min":17.489000000000001},"min":{"1min":5.4859999999999998,"5min":5.4859999999999998,"15min":5.2619999999999996},"max":{"1min":79.078999999999994,"5min":79.078999999999994,"15min":165.99000000000001},"last":7.7380000000000004},{"interface":"front","average":{"1min":20.498000000000001,"5min":18.827000000000002,"15min":22.771999999999998},"min":{"1min":7.508,"5min":6.8140000000000001,"15min":6.7030000000000003},"max":{"1min":117.25,"5min":117.25,"15min":303.892},"last":12.975}]},{"osd":2,"last update":"Sat Nov 25 05:26:18 2023","interfaces":[{"interface":"back","average":{"1min":12.206,"5min":18.088999999999999,"15min":20.989000000000001},"min":{"1min":6.3810000000000002,"5min":6.3810000000000002,"15min":6.3810000000000002},"max":{"1min":23.623999999999999,"5min":169.262,"15min":186.08600000000001},"last":11.487},{"interface":"front","average":{"1min":16.013000000000002,"5min":16.907,"15min":21.454000000000001},"min":{"1min":9.0570000000000004,"5min":7.508,"15min":6.4279999999999999},"max":{"1min":27.609999999999999,"5min":108.26600000000001,"15min":337.41899999999998},"last":9.9469999999999992}]},{"osd":3,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":16.559999999999999,"5min":22.437999999999999,"15min":23.844999999999999},"min":{"1min":12.122999999999999,"5min":8.4469999999999992,"15min":7.5199999999999996},"max":{"1min":28.463000000000001,"5min":152.47399999999999,"15min":238.941},"last":14.534000000000001},{"interface":"front","average":{"1min":18.123000000000001,"5min":21.559999999999999,"15min":26.684999999999999},"min":{"1min":12.221,"5min":6.6929999999999996,"15min":5.5949999999999998},"max":{"1min":29.536000000000001,"5min":150.208,"15min":1292.413},"last":17.239999999999998}]},{"osd":4,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.506,"5min":23.238,"15min":25.268000000000001},"min":{"1min":8.8109999999999999,"5min":8.8109999999999999,"15min":7.9340000000000002},"max":{"1min":21.449999999999999,"5min":120.414,"15min":328.31400000000002},"last":18.576000000000001},{"interface":"front","average":{"1min":27.879000000000001,"5min":21.405000000000001,"15min":23.079999999999998},"min":{"1min":10.739000000000001,"5min":7.1849999999999996,"15min":6.4320000000000004},"max":{"1min":192.684,"5min":192.684,"15min":302.97199999999998},"last":16.614000000000001}]},{"osd":5,"last update":"Sat Nov 25 05:26:03 2023","interfaces":[{"interface":"back","average":{"1min":17.332000000000001,"5min":22.279,"15min":24.291},"min":{"1min":9.1509999999999998,"5min":7.7919999999999998,"15min":7.7619999999999996},"max":{"1min":42.118000000000002,"5min":186.69499999999999,"15min":217.023},"last":19.960000000000001},{"interface":"front","average":{"1min":23.495000000000001,"5min":28.128,"15min":29.045999999999999},"min":{"1min":11.694000000000001,"5min":10.643000000000001,"15min":7.1130000000000004},"max":{"1min":83.798000000000002,"5min":210.22300000000001,"15min":318.11599999999999},"last":17.876999999999999}]},{"osd":6,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.75,"5min":20.798999999999999,"15min":22.933},"min":{"1min":7.069,"5min":6.2380000000000004,"15min":6.2380000000000004},"max":{"1min":100.828,"5min":188.15199999999999,"15min":282.28100000000001},"last":12.269},{"interface":"front","average":{"1min":18.027999999999999,"5min":23.329000000000001,"15min":25.382000000000001},"min":{"1min":8.0459999999999994,"5min":8.0459999999999994,"15min":8.0459999999999994},"max":{"1min":100.143,"5min":187.09899999999999,"15min":215.00700000000001},"last":13.721}]}]},{"osd":5,"up_from":13,"seq":55834575118,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89264,"kb_used_data":13672,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94282576,"statfs":{"total":96636764160,"available":96545357824,"internally_reserved":0,"allocated":14000128,"data_stored":8680375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":17.623000000000001,"5min":14.541,"15min":18.605},"min":{"1min":4.5910000000000002,"5min":4.202,"15min":4.202},"max":{"1min":133.92599999999999,"5min":167.83600000000001,"15min":471.81},"last":6.0540000000000003},{"interface":"front","average":{"1min":14.865,"5min":15.755000000000001,"15min":20.606999999999999},"min":{"1min":6.2699999999999996,"5min":5.4279999999999999,"15min":4.5529999999999999},"max":{"1min":46.905000000000001,"5min":197.834,"15min":317.45699999999999},"last":14.76}]},{"osd":1,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":27.741,"5min":18.356000000000002,"15min":21.565000000000001},"min":{"1min":6.3470000000000004,"5min":5.4189999999999996,"15min":5.0709999999999997},"max":{"1min":140.28999999999999,"5min":140.28999999999999,"15min":263.589},"last":8.7400000000000002},{"interface":"front","average":{"1min":30.024000000000001,"5min":17.751000000000001,"15min":23.059999999999999},"min":{"1min":6.351,"5min":5.1459999999999999,"15min":5.1459999999999999},"max":{"1min":150.196,"5min":150.196,"15min":238.26499999999999},"last":6.9080000000000004}]},{"osd":2,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":19.062999999999999,"5min":19.102,"15min":23.210000000000001},"min":{"1min":7.1890000000000001,"5min":6.2240000000000002,"15min":6.2240000000000002},"max":{"1min":82.295000000000002,"5min":182.477,"15min":416.83199999999999},"last":9.5139999999999993},{"interface":"front","average":{"1min":28.707000000000001,"5min":25.138000000000002,"15min":25.091999999999999},"min":{"1min":12.006,"5min":9.4000000000000004,"15min":7.0880000000000001},"max":{"1min":179.749,"5min":190.517,"15min":207.28700000000001},"last":17.178000000000001}]},{"osd":3,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":31.044,"5min":24.114000000000001,"15min":29.751999999999999},"min":{"1min":10.327,"5min":10.097,"15min":6.6079999999999997},"max":{"1min":166.59700000000001,"5min":166.59700000000001,"15min":443.10300000000001},"last":15.619999999999999},{"interface":"front","average":{"1min":28.024999999999999,"5min":19.308,"15min":31.902000000000001},"min":{"1min":7.851,"5min":6.9800000000000004,"15min":6.2199999999999998},"max":{"1min":130.87299999999999,"5min":130.87299999999999,"15min":1666.6489999999999},"last":11.007}]},{"osd":4,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":30.039000000000001,"5min":25.219999999999999,"15min":25.515999999999998},"min":{"1min":10.957000000000001,"5min":10.657,"15min":9.0280000000000005},"max":{"1min":173.041,"5min":173.041,"15min":173.041},"last":13.186},{"interface":"front","average":{"1min":17.225999999999999,"5min":17.971,"15min":21.449000000000002},"min":{"1min":6.5330000000000004,"5min":6.5330000000000004,"15min":6.5330000000000004},"max":{"1min":68.790000000000006,"5min":123.01300000000001,"15min":156.92500000000001},"last":10.439}]},{"osd":6,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":26.797999999999998,"5min":26.663,"15min":31.244},"min":{"1min":13.439,"5min":11.666,"15min":9.9979999999999993},"max":{"1min":100.47799999999999,"5min":229.86199999999999,"15min":229.86199999999999},"last":13.964},{"interface":"front","average":{"1min":31.841999999999999,"5min":24.629000000000001,"15min":27.375},"min":{"1min":8.6280000000000001,"5min":8.1159999999999997,"15min":7.4770000000000003},"max":{"1min":165.46000000000001,"5min":234.31100000000001,"15min":234.31100000000001},"last":12.1}]},{"osd":7,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":25.68,"5min":20.594999999999999,"15min":26.533999999999999},"min":{"1min":7.2679999999999998,"5min":6.2370000000000001,"15min":6.2370000000000001},"max":{"1min":67.870999999999995,"5min":195.94399999999999,"15min":195.94399999999999},"last":16.361999999999998},{"interface":"front","average":{"1min":23.308,"5min":25.666,"15min":26.741},"min":{"1min":11.151,"5min":11.151,"15min":7.3289999999999997},"max":{"1min":57.329000000000001,"5min":185.74700000000001,"15min":224.99600000000001},"last":11.542999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-25T05:26:48.651 INFO:tasks.ceph:pgid 9.15 last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:48.651 INFO:tasks.ceph:pgid 9.d last_scrub_stamp 2023-11-25T05:05:36.467803+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=5, tm_sec=36, tm_wday=5, tm_yday=329, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=25, tm_hour=5, tm_min=26, tm_sec=19, tm_wday=5, tm_yday=329, tm_isdst=0) 2023-11-25T05:26:48.652 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-11-25T05:26:56.550 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:38.749 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:56.552 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:38.751 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:56.564 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:38.763 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:56.598 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:38.796 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:56.598 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:38.743 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:56.601 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:38.745 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:56.602 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:38.800 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:56.616 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:38.760 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:26:56.619 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:38.817 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:06.574 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:48.772 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:06.579 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:48.777 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:06.595 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:48.793 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:06.629 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:48.773 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:06.630 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:48.774 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:06.643 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:48.787 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:06.648 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:48.846 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:06.650 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:48.848 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:06.667 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:48.865 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:08.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-25T05:27:08.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.765+0000 7ff144ae7640 10 monclient: get_monmap_and_config 2023-11-25T05:27:08.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.765+0000 7ff144ae7640 10 monclient: build_initial_monmap 2023-11-25T05:27:08.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.765+0000 7ff144ae7640 10 monclient: monmap: 2023-11-25T05:27:08.766 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:27:08.766 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:27:08.767 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:27:08.766114+0000 2023-11-25T05:27:08.767 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:27:08.766114+0000 2023-11-25T05:27:08.767 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:27:08.767 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:27:08.767 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:27:08.767 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:27:08.767 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:27:08.767 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:27:08.767 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:27:08.767 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:27:08.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.765+0000 7ff144ae7640 1 Processor -- start 2023-11-25T05:27:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.765+0000 7ff144ae7640 1 -- start start 2023-11-25T05:27:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.765+0000 7ff144ae7640 10 monclient: init 2023-11-25T05:27:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.766+0000 7ff144ae7640 10 monclient: _reopen_session rank -1 2023-11-25T05:27:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.766+0000 7ff144ae7640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2023-11-25T05:27:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.766+0000 7ff144ae7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff1400a8910 addr v1:172.21.15.7:6789/0 2023-11-25T05:27:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.766+0000 7ff144ae7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff1400a9b70 0x7ff14013f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:27:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.766+0000 7ff144ae7640 10 monclient(hunting): picked mon.noname-b con 0x7ff1400a9b70 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:27:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.766+0000 7ff144ae7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff140141f90 0x7ff1401423b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:27:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.766+0000 7ff144ae7640 10 monclient(hunting): picked mon.noname-a con 0x7ff140141f90 addr v2:172.21.15.7:3300/0 2023-11-25T05:27:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.766+0000 7ff144ae7640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1400a8cf0 con 0x7ff1400a8910 2023-11-25T05:27:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.766+0000 7ff144ae7640 10 monclient(hunting): start opening mon connection 2023-11-25T05:27:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.766+0000 7ff144ae7640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1400aabd0 con 0x7ff140141f90 2023-11-25T05:27:08.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.766+0000 7ff144ae7640 10 monclient(hunting): start opening mon connection 2023-11-25T05:27:08.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.766+0000 7ff144ae7640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400aaeb0 con 0x7ff1400a9b70 2023-11-25T05:27:08.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.766+0000 7ff144ae7640 10 monclient(hunting): _renew_subs 2023-11-25T05:27:08.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.766+0000 7ff144ae7640 10 monclient(hunting): authenticate will time out at 2755.009465s 2023-11-25T05:27:08.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.767+0000 7ff13ed9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff140141f90 0x7ff1401423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:27:08.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.767+0000 7ff13dd9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff1400a9b70 0x7ff14013f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:27:08.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.767+0000 7ff13dd9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff1400a9b70 0x7ff14013f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:42288/0 (socket says 172.21.15.7:42288) 2023-11-25T05:27:08.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.767+0000 7ff13dd9b640 1 -- 172.21.15.7:0/2038346628 learned_addr learned my addr 172.21.15.7:0/2038346628 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:27:08.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.767+0000 7ff13ed9d640 10 monclient(hunting): get_auth_request con 0x7ff140141f90 auth_method 0 2023-11-25T05:27:08.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.767+0000 7ff13ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:27:08.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.767+0000 7ff13ed9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:27:08.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.767+0000 7ff13ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:27:08.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.767+0000 7ff13dd9b640 10 monclient(hunting): get_auth_request con 0x7ff1400a9b70 auth_method 0 2023-11-25T05:27:08.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.767+0000 7ff13dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:27:08.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.767+0000 7ff13dd9b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:27:08.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.767+0000 7ff13dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:27:08.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.768+0000 7ff13dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:27:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.768+0000 7ff13dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:27:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.768+0000 7ff13dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:27:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.768+0000 7ff13ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:27:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.768+0000 7ff13ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:27:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.768+0000 7ff13ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:27:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.770+0000 7ff13dd9b640 10 monclient(hunting): handle_auth_done global_id 17453 payload 274 2023-11-25T05:27:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.770+0000 7ff13dd9b640 1 -- 172.21.15.7:0/2038346628 >> v2:172.21.15.7:3300/0 conn(0x7ff140141f90 msgr2=0x7ff1401423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:27:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.770+0000 7ff13dd9b640 1 --2- 172.21.15.7:0/2038346628 >> v2:172.21.15.7:3300/0 conn(0x7ff140141f90 0x7ff1401423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.770+0000 7ff13dd9b640 1 -- 172.21.15.7:0/2038346628 >> v1:172.21.15.7:6789/0 conn(0x7ff1400a8910 legacy=0x7ff14013ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:27:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.770+0000 7ff13dd9b640 10 monclient: _finish_hunting 0 2023-11-25T05:27:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.770+0000 7ff13dd9b640 1 monclient: found mon.noname-b 2023-11-25T05:27:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.770+0000 7ff13dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:27:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.770+0000 7ff13dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:27:08.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.770+0000 7ff13dd9b640 1 -- 172.21.15.7:0/2038346628 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1400a9720 con 0x7ff1400a9b70 2023-11-25T05:27:08.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.770+0000 7ff13ed9d640 1 --2- 172.21.15.7:0/2038346628 >> v2:172.21.15.7:3300/0 conn(0x7ff140141f90 0x7ff1401423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-25T05:27:08.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.772+0000 7ff13dd9b640 1 --2- 172.21.15.7:0/2038346628 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff1400a9b70 0x7ff14013f370 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7ff134009870 tx=0x7ff13402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da4c16b2099f15dc server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:27:08.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff13d59a640 1 -- 172.21.15.7:0/2038346628 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff134031640 con 0x7ff1400a9b70 2023-11-25T05:27:08.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff13d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:27:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff13d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:27:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff13d59a640 10 monclient: dump: 2023-11-25T05:27:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:27:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:27:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:27:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:27:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:27:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:27:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:27:08.778 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:27:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff13d59a640 10 monclient: _finish_auth 0 2023-11-25T05:27:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff13d59a640 10 monclient: _check_auth_tickets 2023-11-25T05:27:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff13d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:27:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff13d59a640 1 -- 172.21.15.7:0/2038346628 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1340317a0 con 0x7ff1400a9b70 2023-11-25T05:27:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff13d59a640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:27:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff13d59a640 1 -- 172.21.15.7:0/2038346628 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff134031b30 con 0x7ff1400a9b70 2023-11-25T05:27:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff13d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:27:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff13d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:27:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff13d59a640 10 monclient: dump: 2023-11-25T05:27:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:27:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:27:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:27:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:27:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:27:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:27:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:27:08.780 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:27:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff144ae7640 5 monclient: authenticate success, global_id 17453 2023-11-25T05:27:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff144ae7640 10 monclient: get_monmap_and_config success 2023-11-25T05:27:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff144ae7640 4 set_mon_vals no callback set 2023-11-25T05:27:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff144ae7640 10 monclient: shutdown 2023-11-25T05:27:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff144ae7640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:27:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff144ae7640 1 -- 172.21.15.7:0/2038346628 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff1400a9b70 msgr2=0x7ff14013f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:27:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.776+0000 7ff144ae7640 1 --2- 172.21.15.7:0/2038346628 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff1400a9b70 0x7ff14013f370 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7ff134009870 tx=0x7ff13402b8f0 comp rx=0 tx=0).stop 2023-11-25T05:27:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.777+0000 7ff144ae7640 1 -- 172.21.15.7:0/2038346628 shutdown_connections 2023-11-25T05:27:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.777+0000 7ff144ae7640 1 --2- 172.21.15.7:0/2038346628 >> v2:172.21.15.7:3300/0 conn(0x7ff140141f90 0x7ff1401423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.777+0000 7ff144ae7640 1 --2- 172.21.15.7:0/2038346628 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff1400a9b70 0x7ff14013f370 unknown :-1 s=CLOSED pgs=3078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.777+0000 7ff144ae7640 1 -- 172.21.15.7:0/2038346628 >> 172.21.15.7:0/2038346628 conn(0x7ff1400b7780 msgr2=0x7ff1400b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:27:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.777+0000 7ff144ae7640 1 -- 172.21.15.7:0/2038346628 shutdown_connections 2023-11-25T05:27:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.777+0000 7ff144ae7640 1 -- 172.21.15.7:0/2038346628 wait complete. 2023-11-25T05:27:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.777+0000 7ff144ae7640 10 monclient: build_initial_monmap 2023-11-25T05:27:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.777+0000 7ff144ae7640 10 monclient: monmap: 2023-11-25T05:27:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:27:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:27:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:27:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:27:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:27:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:27:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:27:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:27:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:27:08.785 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:27:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 1 librados: starting msgr at 2023-11-25T05:27:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 1 librados: starting objecter 2023-11-25T05:27:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 1 Processor -- start 2023-11-25T05:27:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 1 -- start start 2023-11-25T05:27:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 1 librados: setting wanted keys 2023-11-25T05:27:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 1 librados: calling monclient init 2023-11-25T05:27:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 10 monclient: init 2023-11-25T05:27:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 10 monclient: _reopen_session rank -1 2023-11-25T05:27:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-25T05:27:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1400a8910 0x7ff1401391e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:27:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 10 monclient(hunting): picked mon.noname-a con 0x7ff1400a8910 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:27:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff140141f90 0x7ff14013b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:27:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 10 monclient(hunting): picked mon.noname-b con 0x7ff140141f90 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:27:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff14013be80 0x7ff1401da570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:27:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 10 monclient(hunting): picked mon.noname-c con 0x7ff14013be80 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:27:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 10 monclient(hunting): start opening mon connection 2023-11-25T05:27:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff140135b10 con 0x7ff1400a8910 2023-11-25T05:27:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 10 monclient(hunting): start opening mon connection 2023-11-25T05:27:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff140139f40 con 0x7ff140141f90 2023-11-25T05:27:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 10 monclient(hunting): start opening mon connection 2023-11-25T05:27:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14013af60 con 0x7ff14013be80 2023-11-25T05:27:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 10 monclient(hunting): _renew_subs 2023-11-25T05:27:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.778+0000 7ff144ae7640 10 monclient(hunting): authenticate will time out at 2755.021768s 2023-11-25T05:27:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.779+0000 7ff13e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1400a8910 0x7ff1401391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:27:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.779+0000 7ff13ed9d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff14013be80 0x7ff1401da570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:27:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.779+0000 7ff13dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff140141f90 0x7ff14013b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:27:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.779+0000 7ff13e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1400a8910 0x7ff1401391e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47632/0 (socket says 172.21.15.7:47632) 2023-11-25T05:27:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.779+0000 7ff13e59c640 1 -- 172.21.15.7:0/3315895629 learned_addr learned my addr 172.21.15.7:0/3315895629 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:27:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.779+0000 7ff13e59c640 10 monclient(hunting): get_auth_request con 0x7ff1400a8910 auth_method 0 2023-11-25T05:27:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.779+0000 7ff13e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:27:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.779+0000 7ff13e59c640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:27:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.779+0000 7ff13e59c640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:27:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.779+0000 7ff13ed9d640 10 monclient(hunting): get_auth_request con 0x7ff14013be80 auth_method 0 2023-11-25T05:27:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.779+0000 7ff13ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:27:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.779+0000 7ff13ed9d640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:27:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.779+0000 7ff13ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:27:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.780+0000 7ff13dd9b640 10 monclient(hunting): get_auth_request con 0x7ff140141f90 auth_method 0 2023-11-25T05:27:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.780+0000 7ff13dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:27:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.780+0000 7ff13dd9b640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:27:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.780+0000 7ff13dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:27:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.780+0000 7ff13e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:27:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.780+0000 7ff13e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:27:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.780+0000 7ff13e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:27:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.780+0000 7ff13ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:27:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.780+0000 7ff13ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:27:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.780+0000 7ff13ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:27:08.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.780+0000 7ff13dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:27:08.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.780+0000 7ff13dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:27:08.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.780+0000 7ff13dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:27:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.783+0000 7ff13dd9b640 10 monclient(hunting): handle_auth_done global_id 21319 payload 824 2023-11-25T05:27:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.783+0000 7ff13dd9b640 1 -- 172.21.15.7:0/3315895629 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff14013be80 msgr2=0x7ff1401da570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:27:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.783+0000 7ff13dd9b640 1 --2- 172.21.15.7:0/3315895629 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff14013be80 0x7ff1401da570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.783+0000 7ff13dd9b640 1 -- 172.21.15.7:0/3315895629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1400a8910 msgr2=0x7ff1401391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:27:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.783+0000 7ff13dd9b640 1 --2- 172.21.15.7:0/3315895629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1400a8910 0x7ff1401391e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.783+0000 7ff13dd9b640 10 monclient: _finish_hunting 0 2023-11-25T05:27:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.783+0000 7ff13dd9b640 1 monclient: found mon.noname-b 2023-11-25T05:27:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.783+0000 7ff13dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:27:08.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.783+0000 7ff13dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:27:08.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.783+0000 7ff13dd9b640 1 -- 172.21.15.7:0/3315895629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff138009400 con 0x7ff140141f90 2023-11-25T05:27:08.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.784+0000 7ff13e59c640 1 --2- 172.21.15.7:0/3315895629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1400a8910 0x7ff1401391e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-25T05:27:08.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.786+0000 7ff13dd9b640 1 --2- 172.21.15.7:0/3315895629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff140141f90 0x7ff14013b940 secure :-1 s=READY pgs=5177 cs=0 l=1 rev1=1 crypto rx=0x7ff134033000 tx=0x7ff1340046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:27:08.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.786+0000 7ff12effd640 10 client.?.objecter ms_handle_connect 0x7ff140141f90 2023-11-25T05:27:08.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.786+0000 7ff12effd640 10 client.?.objecter resend_mon_ops 2023-11-25T05:27:08.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.790+0000 7ff12effd640 1 -- 172.21.15.7:0/3315895629 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13403ac60 con 0x7ff140141f90 2023-11-25T05:27:08.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.790+0000 7ff12effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:27:08.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.790+0000 7ff12effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:27:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.790+0000 7ff12effd640 10 monclient: dump: 2023-11-25T05:27:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:27:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:27:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:27:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:27:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:27:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:27:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:27:08.793 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:27:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.790+0000 7ff12effd640 10 monclient: _finish_auth 0 2023-11-25T05:27:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.790+0000 7ff12effd640 10 monclient: _check_auth_tickets 2023-11-25T05:27:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.790+0000 7ff12effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:27:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.790+0000 7ff144ae7640 5 monclient: authenticate success, global_id 21319 2023-11-25T05:27:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.790+0000 7ff12effd640 1 -- 172.21.15.7:0/3315895629 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13403adc0 con 0x7ff140141f90 2023-11-25T05:27:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.790+0000 7ff144ae7640 10 monclient: _renew_subs 2023-11-25T05:27:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.790+0000 7ff144ae7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:27:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.790+0000 7ff144ae7640 1 -- 172.21.15.7:0/3315895629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1340094d0 con 0x7ff140141f90 2023-11-25T05:27:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.791+0000 7ff12effd640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:27:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.791+0000 7ff12effd640 1 -- 172.21.15.7:0/3315895629 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff134044630 con 0x7ff140141f90 2023-11-25T05:27:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.791+0000 7ff12effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:27:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.791+0000 7ff12effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:27:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.791+0000 7ff12effd640 10 monclient: dump: 2023-11-25T05:27:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:27:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:27:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:27:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:27:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:27:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:27:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:27:08.795 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:27:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.791+0000 7ff13cd99640 4 set_mon_vals no callback set 2023-11-25T05:27:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.791+0000 7ff144ae7640 10 client.21319.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:27:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.791+0000 7ff144ae7640 10 monclient: _renew_subs 2023-11-25T05:27:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.791+0000 7ff144ae7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:27:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.791+0000 7ff144ae7640 1 -- 172.21.15.7:0/3315895629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14013b520 con 0x7ff140141f90 2023-11-25T05:27:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.791+0000 7ff144ae7640 1 librados: init done 2023-11-25T05:27:08.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.791+0000 7ff144ae7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:27:08.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.792+0000 7ff144ae7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:27:08.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.792+0000 7ff144ae7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:27:08.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.792+0000 7ff144ae7640 1 -- 172.21.15.7:0/3315895629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff140075b60 con 0x7ff140141f90 2023-11-25T05:27:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.813+0000 7ff12effd640 1 -- 172.21.15.7:0/3315895629 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff13402f050 con 0x7ff140141f90 2023-11-25T05:27:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.813+0000 7ff12effd640 1 --2- 172.21.15.7:0/3315895629 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ff12006c820 0x7ff12006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:27:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.813+0000 7ff13e59c640 1 --2- 172.21.15.7:0/3315895629 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ff12006c820 0x7ff12006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:27:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.813+0000 7ff13e59c640 10 monclient: get_auth_request con 0x7ff12006c820 auth_method 0 2023-11-25T05:27:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.814+0000 7ff12effd640 10 client.21319.objecter ms_handle_connect 0x7ff12006c820 2023-11-25T05:27:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.814+0000 7ff13e59c640 1 --2- 172.21.15.7:0/3315895629 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ff12006c820 0x7ff12006ece0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff130007fd0 tx=0x7ff130009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:27:08.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.816+0000 7ff12effd640 1 -- 172.21.15.7:0/3315895629 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 687..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7ff13407aca0 con 0x7ff140141f90 2023-11-25T05:27:08.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.816+0000 7ff12effd640 10 client.21319.objecter ms_dispatch 0x7ff140000b90 osd_map(918..918 src has 687..918) v4 2023-11-25T05:27:08.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.816+0000 7ff12effd640 3 client.21319.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:27:08.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.816+0000 7ff12effd640 3 client.21319.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:27:08.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:08.816+0000 7ff12effd640 20 client.21319.objecter dump_active .. 0 homeless 2023-11-25T05:27:09.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.047+0000 7ff12effd640 1 -- 172.21.15.7:0/3315895629 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1340768e0 con 0x7ff140141f90 2023-11-25T05:27:09.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.047+0000 7ff12effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:27:09.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.047+0000 7ff12effd640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:27:09.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.223+0000 7ff144ae7640 1 -- 172.21.15.7:0/3315895629 --> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff1400b4ea0 con 0x7ff12006c820 2023-11-25T05:27:09.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.228+0000 7ff12effd640 1 -- 172.21.15.7:0/3315895629 <== mgr.4110 v2:172.21.15.90:6800/1055092586 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152101 (secure 0 0 0) 0x7ff1400b4ea0 con 0x7ff12006c820 2023-11-25T05:27:09.231 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-25T05:27:09.232 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-25T05:27:09.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 10 librados: watch_flush enter 2023-11-25T05:27:09.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 10 librados: watch_flush exit 2023-11-25T05:27:09.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 20 client.21319.objecter shutdown clearing up homeless session... 2023-11-25T05:27:09.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 10 client.21319.objecter successfully canceled tick 2023-11-25T05:27:09.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 1 -- 172.21.15.7:0/3315895629 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ff12006c820 msgr2=0x7ff12006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:27:09.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 1 --2- 172.21.15.7:0/3315895629 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ff12006c820 0x7ff12006ece0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff130007fd0 tx=0x7ff130009040 comp rx=0 tx=0).stop 2023-11-25T05:27:09.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 10 monclient: shutdown 2023-11-25T05:27:09.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:27:09.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 1 -- 172.21.15.7:0/3315895629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff140141f90 msgr2=0x7ff14013b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:27:09.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 1 --2- 172.21.15.7:0/3315895629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff140141f90 0x7ff14013b940 secure :-1 s=READY pgs=5177 cs=0 l=1 rev1=1 crypto rx=0x7ff134033000 tx=0x7ff1340046c0 comp rx=0 tx=0).stop 2023-11-25T05:27:09.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 1 -- 172.21.15.7:0/3315895629 shutdown_connections 2023-11-25T05:27:09.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 1 --2- 172.21.15.7:0/3315895629 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7ff12006c820 0x7ff12006ece0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:09.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 1 --2- 172.21.15.7:0/3315895629 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff14013be80 0x7ff1401da570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:09.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 1 --2- 172.21.15.7:0/3315895629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff140141f90 0x7ff14013b940 unknown :-1 s=CLOSED pgs=5177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:09.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 1 --2- 172.21.15.7:0/3315895629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1400a8910 0x7ff1401391e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:09.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 1 -- 172.21.15.7:0/3315895629 >> 172.21.15.7:0/3315895629 conn(0x7ff1400b7780 msgr2=0x7ff14013f180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:27:09.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 1 -- 172.21.15.7:0/3315895629 shutdown_connections 2023-11-25T05:27:09.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 1 -- 172.21.15.7:0/3315895629 wait complete. 2023-11-25T05:27:09.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.230+0000 7ff144ae7640 1 librados: shutdown 2023-11-25T05:27:09.244 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1605,"stamp":"2023-11-25T05:27:07.725456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":596568,"kb_used_data":107032,"kb_used_omap":12,"kb_used_meta":489459,"kb_avail":754378152,"statfs":{"total":773094113280,"available":772483227648,"internally_reserved":0,"allocated":109600768,"data_stored":67055970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":501206579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003017"},"pg_stats":[{"pgid":"9.15","version":"0'0","reported_seq":870,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:48.426726+0000","last_change":"2023-11-25T05:26:48.427458+0000","last_active":"2023-11-25T05:26:48.426726+0000","last_peered":"2023-11-25T05:26:48.426726+0000","last_clean":"2023-11-25T05:26:48.426726+0000","last_became_active":"2023-11-25T05:05:46.875394+0000","last_became_peered":"2023-11-25T05:05:46.875394+0000","last_unstale":"2023-11-25T05:26:48.426726+0000","last_undegraded":"2023-11-25T05:26:48.426726+0000","last_fullsized":"2023-11-25T05:26:48.426726+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:48.425348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:48.425348+0000","last_clean_scrub_stamp":"2023-11-25T05:26:48.425348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:41:41.404122+0000","scrub_duration":0.042045065999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:38.240800+0000","last_change":"2023-11-25T05:26:38.240800+0000","last_active":"2023-11-25T05:26:38.240800+0000","last_peered":"2023-11-25T05:26:38.240800+0000","last_clean":"2023-11-25T05:26:38.240800+0000","last_became_active":"2023-11-25T05:05:46.923663+0000","last_became_peered":"2023-11-25T05:05:46.923663+0000","last_unstale":"2023-11-25T05:26:38.240800+0000","last_undegraded":"2023-11-25T05:26:38.240800+0000","last_fullsized":"2023-11-25T05:26:38.240800+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:38.238720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:38.238720+0000","last_clean_scrub_stamp":"2023-11-25T05:26:38.238720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:16:48.540111+0000","scrub_duration":0.031357639,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:26.249719+0000","last_change":"2023-11-25T05:26:26.250999+0000","last_active":"2023-11-25T05:26:26.249719+0000","last_peered":"2023-11-25T05:26:26.249719+0000","last_clean":"2023-11-25T05:26:26.249719+0000","last_became_active":"2023-11-25T05:05:50.968719+0000","last_became_peered":"2023-11-25T05:05:50.968719+0000","last_unstale":"2023-11-25T05:26:26.249719+0000","last_undegraded":"2023-11-25T05:26:26.249719+0000","last_fullsized":"2023-11-25T05:26:26.249719+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:26.244542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:26.244542+0000","last_clean_scrub_stamp":"2023-11-25T05:26:26.244542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:07:20.535214+0000","scrub_duration":0.16470911399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":860,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:22.444189+0000","last_change":"2023-11-25T05:26:22.444916+0000","last_active":"2023-11-25T05:26:22.444189+0000","last_peered":"2023-11-25T05:26:22.444189+0000","last_clean":"2023-11-25T05:26:22.444189+0000","last_became_active":"2023-11-25T05:06:05.084111+0000","last_became_peered":"2023-11-25T05:06:05.084111+0000","last_unstale":"2023-11-25T05:26:22.444189+0000","last_undegraded":"2023-11-25T05:26:22.444189+0000","last_fullsized":"2023-11-25T05:26:22.444189+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:22.439964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:22.439964+0000","last_clean_scrub_stamp":"2023-11-25T05:26:22.439964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:31:36.581238+0000","scrub_duration":0.053157549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":871,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:46.364433+0000","last_change":"2023-11-25T05:26:46.365167+0000","last_active":"2023-11-25T05:26:46.364433+0000","last_peered":"2023-11-25T05:26:46.364433+0000","last_clean":"2023-11-25T05:26:46.364433+0000","last_became_active":"2023-11-25T05:05:47.855679+0000","last_became_peered":"2023-11-25T05:05:47.855679+0000","last_unstale":"2023-11-25T05:26:46.364433+0000","last_undegraded":"2023-11-25T05:26:46.364433+0000","last_fullsized":"2023-11-25T05:26:46.364433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:46.363023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:46.363023+0000","last_clean_scrub_stamp":"2023-11-25T05:26:46.363023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:29:33.832022+0000","scrub_duration":0.041375328000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":886,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:34.216155+0000","last_change":"2023-11-25T05:26:34.216155+0000","last_active":"2023-11-25T05:26:34.216155+0000","last_peered":"2023-11-25T05:26:34.216155+0000","last_clean":"2023-11-25T05:26:34.216155+0000","last_became_active":"2023-11-25T05:06:09.633454+0000","last_became_peered":"2023-11-25T05:06:09.633454+0000","last_unstale":"2023-11-25T05:26:34.216155+0000","last_undegraded":"2023-11-25T05:26:34.216155+0000","last_fullsized":"2023-11-25T05:26:34.216155+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:34.214048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:34.214048+0000","last_clean_scrub_stamp":"2023-11-25T05:26:34.214048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:15:13.883268+0000","scrub_duration":0.045256073000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:23.194252+0000","last_change":"2023-11-25T05:26:23.194252+0000","last_active":"2023-11-25T05:26:23.194252+0000","last_peered":"2023-11-25T05:26:23.194252+0000","last_clean":"2023-11-25T05:26:23.194252+0000","last_became_active":"2023-11-25T05:05:53.099253+0000","last_became_peered":"2023-11-25T05:05:53.099253+0000","last_unstale":"2023-11-25T05:26:23.194252+0000","last_undegraded":"2023-11-25T05:26:23.194252+0000","last_fullsized":"2023-11-25T05:26:23.194252+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:23.192030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:23.192030+0000","last_clean_scrub_stamp":"2023-11-25T05:26:23.192030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T14:46:00.120705+0000","scrub_duration":0.056747789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":886,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:29.174553+0000","last_change":"2023-11-25T05:26:29.174553+0000","last_active":"2023-11-25T05:26:29.174553+0000","last_peered":"2023-11-25T05:26:29.174553+0000","last_clean":"2023-11-25T05:26:29.174553+0000","last_became_active":"2023-11-25T05:05:43.496305+0000","last_became_peered":"2023-11-25T05:05:43.496305+0000","last_unstale":"2023-11-25T05:26:29.174553+0000","last_undegraded":"2023-11-25T05:26:29.174553+0000","last_fullsized":"2023-11-25T05:26:29.174553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:29.172138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:29.172138+0000","last_clean_scrub_stamp":"2023-11-25T05:26:29.172138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:43:52.822726+0000","scrub_duration":0.038697677,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":881,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:33.893713+0000","last_change":"2023-11-25T05:26:33.894594+0000","last_active":"2023-11-25T05:26:33.893713+0000","last_peered":"2023-11-25T05:26:33.893713+0000","last_clean":"2023-11-25T05:26:33.893713+0000","last_became_active":"2023-11-25T05:05:42.672388+0000","last_became_peered":"2023-11-25T05:05:42.672388+0000","last_unstale":"2023-11-25T05:26:33.893713+0000","last_undegraded":"2023-11-25T05:26:33.893713+0000","last_fullsized":"2023-11-25T05:26:33.893713+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:33.892390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:33.892390+0000","last_clean_scrub_stamp":"2023-11-25T05:26:33.892390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:32:42.524686+0000","scrub_duration":0.077926995999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:43.208623+0000","last_change":"2023-11-25T05:26:43.209282+0000","last_active":"2023-11-25T05:26:43.208623+0000","last_peered":"2023-11-25T05:26:43.208623+0000","last_clean":"2023-11-25T05:26:43.208623+0000","last_became_active":"2023-11-25T05:05:46.959854+0000","last_became_peered":"2023-11-25T05:05:46.959854+0000","last_unstale":"2023-11-25T05:26:43.208623+0000","last_undegraded":"2023-11-25T05:26:43.208623+0000","last_fullsized":"2023-11-25T05:26:43.208623+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:43.207305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:43.207305+0000","last_clean_scrub_stamp":"2023-11-25T05:26:43.207305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:31:56.183159+0000","scrub_duration":0.031291102000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:30.893207+0000","last_change":"2023-11-25T05:26:30.893854+0000","last_active":"2023-11-25T05:26:30.893207+0000","last_peered":"2023-11-25T05:26:30.893207+0000","last_clean":"2023-11-25T05:26:30.893207+0000","last_became_active":"2023-11-25T05:05:47.753176+0000","last_became_peered":"2023-11-25T05:05:47.753176+0000","last_unstale":"2023-11-25T05:26:30.893207+0000","last_undegraded":"2023-11-25T05:26:30.893207+0000","last_fullsized":"2023-11-25T05:26:30.893207+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:30.891827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:30.891827+0000","last_clean_scrub_stamp":"2023-11-25T05:26:30.891827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:13:27.472572+0000","scrub_duration":0.076990036999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:35.832685+0000","last_change":"2023-11-25T05:26:35.833330+0000","last_active":"2023-11-25T05:26:35.832685+0000","last_peered":"2023-11-25T05:26:35.832685+0000","last_clean":"2023-11-25T05:26:35.832685+0000","last_became_active":"2023-11-25T05:05:45.597733+0000","last_became_peered":"2023-11-25T05:05:45.597733+0000","last_unstale":"2023-11-25T05:26:35.832685+0000","last_undegraded":"2023-11-25T05:26:35.832685+0000","last_fullsized":"2023-11-25T05:26:35.832685+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:35.831376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:35.831376+0000","last_clean_scrub_stamp":"2023-11-25T05:26:35.831376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T13:25:11.746685+0000","scrub_duration":0.061482579000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":876,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:33.098498+0000","last_change":"2023-11-25T05:26:33.099152+0000","last_active":"2023-11-25T05:26:33.098498+0000","last_peered":"2023-11-25T05:26:33.098498+0000","last_clean":"2023-11-25T05:26:33.098498+0000","last_became_active":"2023-11-25T05:05:47.792727+0000","last_became_peered":"2023-11-25T05:05:47.792727+0000","last_unstale":"2023-11-25T05:26:33.098498+0000","last_undegraded":"2023-11-25T05:26:33.098498+0000","last_fullsized":"2023-11-25T05:26:33.098498+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:33.097199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:33.097199+0000","last_clean_scrub_stamp":"2023-11-25T05:26:33.097199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:10:01.913154+0000","scrub_duration":0.034383390999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:44.447839+0000","last_change":"2023-11-25T05:26:44.448585+0000","last_active":"2023-11-25T05:26:44.447839+0000","last_peered":"2023-11-25T05:26:44.447839+0000","last_clean":"2023-11-25T05:26:44.447839+0000","last_became_active":"2023-11-25T05:04:33.577510+0000","last_became_peered":"2023-11-25T05:04:33.577510+0000","last_unstale":"2023-11-25T05:26:44.447839+0000","last_undegraded":"2023-11-25T05:26:44.447839+0000","last_fullsized":"2023-11-25T05:26:44.447839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:44.446380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:44.446380+0000","last_clean_scrub_stamp":"2023-11-25T05:26:44.446380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:50:58.487234+0000","scrub_duration":0.039103001999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":871,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:53.383567+0000","last_change":"2023-11-25T05:26:53.384312+0000","last_active":"2023-11-25T05:26:53.383567+0000","last_peered":"2023-11-25T05:26:53.383567+0000","last_clean":"2023-11-25T05:26:53.383567+0000","last_became_active":"2023-11-25T05:05:58.695110+0000","last_became_peered":"2023-11-25T05:05:58.695110+0000","last_unstale":"2023-11-25T05:26:53.383567+0000","last_undegraded":"2023-11-25T05:26:53.383567+0000","last_fullsized":"2023-11-25T05:26:53.383567+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:53.379993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:53.379993+0000","last_clean_scrub_stamp":"2023-11-25T05:26:53.379993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:37:09.448728+0000","scrub_duration":0.059050598000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":886,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:26.960188+0000","last_change":"2023-11-25T05:26:26.961299+0000","last_active":"2023-11-25T05:26:26.960188+0000","last_peered":"2023-11-25T05:26:26.960188+0000","last_clean":"2023-11-25T05:26:26.960188+0000","last_became_active":"2023-11-25T05:04:33.610776+0000","last_became_peered":"2023-11-25T05:04:33.610776+0000","last_unstale":"2023-11-25T05:26:26.960188+0000","last_undegraded":"2023-11-25T05:26:26.960188+0000","last_fullsized":"2023-11-25T05:26:26.960188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:26.958130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:26.958130+0000","last_clean_scrub_stamp":"2023-11-25T05:26:26.958130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:07:22.646353+0000","scrub_duration":0.053590394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:29.092414+0000","last_change":"2023-11-25T05:26:29.093149+0000","last_active":"2023-11-25T05:26:29.092414+0000","last_peered":"2023-11-25T05:26:29.092414+0000","last_clean":"2023-11-25T05:26:29.092414+0000","last_became_active":"2023-11-25T05:06:12.543810+0000","last_became_peered":"2023-11-25T05:06:12.543810+0000","last_unstale":"2023-11-25T05:26:29.092414+0000","last_undegraded":"2023-11-25T05:26:29.092414+0000","last_fullsized":"2023-11-25T05:26:29.092414+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:29.091057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:29.091057+0000","last_clean_scrub_stamp":"2023-11-25T05:26:29.091057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:56:22.665568+0000","scrub_duration":0.035620111000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":876,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:40.457222+0000","last_change":"2023-11-25T05:26:40.457946+0000","last_active":"2023-11-25T05:26:40.457222+0000","last_peered":"2023-11-25T05:26:40.457222+0000","last_clean":"2023-11-25T05:26:40.457222+0000","last_became_active":"2023-11-25T05:04:33.337158+0000","last_became_peered":"2023-11-25T05:04:33.337158+0000","last_unstale":"2023-11-25T05:26:40.457222+0000","last_undegraded":"2023-11-25T05:26:40.457222+0000","last_fullsized":"2023-11-25T05:26:40.457222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:40.455774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:40.455774+0000","last_clean_scrub_stamp":"2023-11-25T05:26:40.455774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:48:38.148988+0000","scrub_duration":0.032964660999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:27.153493+0000","last_change":"2023-11-25T05:26:27.154234+0000","last_active":"2023-11-25T05:26:27.153493+0000","last_peered":"2023-11-25T05:26:27.153493+0000","last_clean":"2023-11-25T05:26:27.153493+0000","last_became_active":"2023-11-25T05:05:51.097150+0000","last_became_peered":"2023-11-25T05:05:51.097150+0000","last_unstale":"2023-11-25T05:26:27.153493+0000","last_undegraded":"2023-11-25T05:26:27.153493+0000","last_fullsized":"2023-11-25T05:26:27.153493+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:27.152003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:27.152003+0000","last_clean_scrub_stamp":"2023-11-25T05:26:27.152003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T13:29:44.174699+0000","scrub_duration":0.074299855999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":885,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:26.311099+0000","last_change":"2023-11-25T05:26:26.315627+0000","last_active":"2023-11-25T05:26:26.311099+0000","last_peered":"2023-11-25T05:26:26.311099+0000","last_clean":"2023-11-25T05:26:26.311099+0000","last_became_active":"2023-11-25T05:04:33.388849+0000","last_became_peered":"2023-11-25T05:04:33.388849+0000","last_unstale":"2023-11-25T05:26:26.311099+0000","last_undegraded":"2023-11-25T05:26:26.311099+0000","last_fullsized":"2023-11-25T05:26:26.311099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:26.307049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:26.307049+0000","last_clean_scrub_stamp":"2023-11-25T05:26:26.307049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:30:19.433152+0000","scrub_duration":0.21619841400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:31.193834+0000","last_change":"2023-11-25T05:26:31.194481+0000","last_active":"2023-11-25T05:26:31.193834+0000","last_peered":"2023-11-25T05:26:31.193834+0000","last_clean":"2023-11-25T05:26:31.193834+0000","last_became_active":"2023-11-25T05:05:55.038711+0000","last_became_peered":"2023-11-25T05:05:55.038711+0000","last_unstale":"2023-11-25T05:26:31.193834+0000","last_undegraded":"2023-11-25T05:26:31.193834+0000","last_fullsized":"2023-11-25T05:26:31.193834+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:31.192518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:31.192518+0000","last_clean_scrub_stamp":"2023-11-25T05:26:31.192518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:23:23.245247+0000","scrub_duration":0.042831777000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":876,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:29.343188+0000","last_change":"2023-11-25T05:26:29.343892+0000","last_active":"2023-11-25T05:26:29.343188+0000","last_peered":"2023-11-25T05:26:29.343188+0000","last_clean":"2023-11-25T05:26:29.343188+0000","last_became_active":"2023-11-25T05:04:33.376375+0000","last_became_peered":"2023-11-25T05:04:33.376375+0000","last_unstale":"2023-11-25T05:26:29.343188+0000","last_undegraded":"2023-11-25T05:26:29.343188+0000","last_fullsized":"2023-11-25T05:26:29.343188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:29.341842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:29.341842+0000","last_clean_scrub_stamp":"2023-11-25T05:26:29.341842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:20:11.366575+0000","scrub_duration":0.028595209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":887,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:30.155622+0000","last_change":"2023-11-25T05:26:30.156324+0000","last_active":"2023-11-25T05:26:30.155622+0000","last_peered":"2023-11-25T05:26:30.155622+0000","last_clean":"2023-11-25T05:26:30.155622+0000","last_became_active":"2023-11-25T05:05:44.554983+0000","last_became_peered":"2023-11-25T05:05:44.554983+0000","last_unstale":"2023-11-25T05:26:30.155622+0000","last_undegraded":"2023-11-25T05:26:30.155622+0000","last_fullsized":"2023-11-25T05:26:30.155622+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:30.154261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:30.154261+0000","last_clean_scrub_stamp":"2023-11-25T05:26:30.154261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:53:16.053380+0000","scrub_duration":0.054855872999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:28.100253+0000","last_change":"2023-11-25T05:26:28.100948+0000","last_active":"2023-11-25T05:26:28.100253+0000","last_peered":"2023-11-25T05:26:28.100253+0000","last_clean":"2023-11-25T05:26:28.100253+0000","last_became_active":"2023-11-25T05:04:33.196533+0000","last_became_peered":"2023-11-25T05:04:33.196533+0000","last_unstale":"2023-11-25T05:26:28.100253+0000","last_undegraded":"2023-11-25T05:26:28.100253+0000","last_fullsized":"2023-11-25T05:26:28.100253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:28.098831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:28.098831+0000","last_clean_scrub_stamp":"2023-11-25T05:26:28.098831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T13:26:33.808201+0000","scrub_duration":0.031799603000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:36.233927+0000","last_change":"2023-11-25T05:26:36.234604+0000","last_active":"2023-11-25T05:26:36.233927+0000","last_peered":"2023-11-25T05:26:36.233927+0000","last_clean":"2023-11-25T05:26:36.233927+0000","last_became_active":"2023-11-25T05:05:45.759824+0000","last_became_peered":"2023-11-25T05:05:45.759824+0000","last_unstale":"2023-11-25T05:26:36.233927+0000","last_undegraded":"2023-11-25T05:26:36.233927+0000","last_fullsized":"2023-11-25T05:26:36.233927+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:36.232550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:36.232550+0000","last_clean_scrub_stamp":"2023-11-25T05:26:36.232550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T09:59:53.238852+0000","scrub_duration":0.081003382999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":877,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:27.343406+0000","last_change":"2023-11-25T05:26:27.343406+0000","last_active":"2023-11-25T05:26:27.343406+0000","last_peered":"2023-11-25T05:26:27.343406+0000","last_clean":"2023-11-25T05:26:27.343406+0000","last_became_active":"2023-11-25T05:04:33.242068+0000","last_became_peered":"2023-11-25T05:04:33.242068+0000","last_unstale":"2023-11-25T05:26:27.343406+0000","last_undegraded":"2023-11-25T05:26:27.343406+0000","last_fullsized":"2023-11-25T05:26:27.343406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:27.341037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:27.341037+0000","last_clean_scrub_stamp":"2023-11-25T05:26:27.341037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:13:18.486294+0000","scrub_duration":0.033598854999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:27.111527+0000","last_change":"2023-11-25T05:26:27.112227+0000","last_active":"2023-11-25T05:26:27.111527+0000","last_peered":"2023-11-25T05:26:27.111527+0000","last_clean":"2023-11-25T05:26:27.111527+0000","last_became_active":"2023-11-25T05:05:45.057546+0000","last_became_peered":"2023-11-25T05:05:45.057546+0000","last_unstale":"2023-11-25T05:26:27.111527+0000","last_undegraded":"2023-11-25T05:26:27.111527+0000","last_fullsized":"2023-11-25T05:26:27.111527+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:27.110149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:27.110149+0000","last_clean_scrub_stamp":"2023-11-25T05:26:27.110149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T17:19:27.768590+0000","scrub_duration":0.035816397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":876,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:37.485974+0000","last_change":"2023-11-25T05:26:37.486693+0000","last_active":"2023-11-25T05:26:37.485974+0000","last_peered":"2023-11-25T05:26:37.485974+0000","last_clean":"2023-11-25T05:26:37.485974+0000","last_became_active":"2023-11-25T05:04:33.778042+0000","last_became_peered":"2023-11-25T05:04:33.778042+0000","last_unstale":"2023-11-25T05:26:37.485974+0000","last_undegraded":"2023-11-25T05:26:37.485974+0000","last_fullsized":"2023-11-25T05:26:37.485974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:37.484485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:37.484485+0000","last_clean_scrub_stamp":"2023-11-25T05:26:37.484485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T11:52:43.582599+0000","scrub_duration":0.033599663000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"918'78","reported_seq":988,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:22.189192+0000","last_change":"2023-11-25T05:26:22.189192+0000","last_active":"2023-11-25T05:26:22.189192+0000","last_peered":"2023-11-25T05:26:22.189192+0000","last_clean":"2023-11-25T05:26:22.189192+0000","last_became_active":"2023-11-25T05:04:27.388156+0000","last_became_peered":"2023-11-25T05:04:27.388156+0000","last_unstale":"2023-11-25T05:26:22.189192+0000","last_undegraded":"2023-11-25T05:26:22.189192+0000","last_fullsized":"2023-11-25T05:26:22.189192+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'78","last_scrub_stamp":"2023-11-25T05:26:22.186221+0000","last_deep_scrub":"918'78","last_deep_scrub_stamp":"2023-11-25T05:26:22.186221+0000","last_clean_scrub_stamp":"2023-11-25T05:26:22.186221+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T15:49:24.005907+0000","scrub_duration":0.15619770899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":883,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:27.854139+0000","last_change":"2023-11-25T05:26:27.854139+0000","last_active":"2023-11-25T05:26:27.854139+0000","last_peered":"2023-11-25T05:26:27.854139+0000","last_clean":"2023-11-25T05:26:27.854139+0000","last_became_active":"2023-11-25T05:05:47.973359+0000","last_became_peered":"2023-11-25T05:05:47.973359+0000","last_unstale":"2023-11-25T05:26:27.854139+0000","last_undegraded":"2023-11-25T05:26:27.854139+0000","last_fullsized":"2023-11-25T05:26:27.854139+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:27.849775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:27.849775+0000","last_clean_scrub_stamp":"2023-11-25T05:26:27.849775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:25:29.815659+0000","scrub_duration":0.026694712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":863,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:21.466727+0000","last_change":"2023-11-25T05:26:21.466727+0000","last_active":"2023-11-25T05:26:21.466727+0000","last_peered":"2023-11-25T05:26:21.466727+0000","last_clean":"2023-11-25T05:26:21.466727+0000","last_became_active":"2023-11-25T05:05:54.045707+0000","last_became_peered":"2023-11-25T05:05:54.045707+0000","last_unstale":"2023-11-25T05:26:21.466727+0000","last_undegraded":"2023-11-25T05:26:21.466727+0000","last_fullsized":"2023-11-25T05:26:21.466727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:21.462123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:21.462123+0000","last_clean_scrub_stamp":"2023-11-25T05:26:21.462123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:08:12.548363+0000","scrub_duration":0.061865305000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":881,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:28.145820+0000","last_change":"2023-11-25T05:26:28.146469+0000","last_active":"2023-11-25T05:26:28.145820+0000","last_peered":"2023-11-25T05:26:28.145820+0000","last_clean":"2023-11-25T05:26:28.145820+0000","last_became_active":"2023-11-25T05:05:42.487757+0000","last_became_peered":"2023-11-25T05:05:42.487757+0000","last_unstale":"2023-11-25T05:26:28.145820+0000","last_undegraded":"2023-11-25T05:26:28.145820+0000","last_fullsized":"2023-11-25T05:26:28.145820+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:28.144553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:28.144553+0000","last_clean_scrub_stamp":"2023-11-25T05:26:28.144553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T09:43:48.401548+0000","scrub_duration":0.033698147999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"918'8","reported_seq":892,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:31.158768+0000","last_change":"2023-11-25T05:26:31.159600+0000","last_active":"2023-11-25T05:26:31.158768+0000","last_peered":"2023-11-25T05:26:31.158768+0000","last_clean":"2023-11-25T05:26:31.158768+0000","last_became_active":"2023-11-25T05:05:45.212870+0000","last_became_peered":"2023-11-25T05:05:45.212870+0000","last_unstale":"2023-11-25T05:26:31.158768+0000","last_undegraded":"2023-11-25T05:26:31.158768+0000","last_fullsized":"2023-11-25T05:26:31.158768+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'8","last_scrub_stamp":"2023-11-25T05:26:31.157416+0000","last_deep_scrub":"918'8","last_deep_scrub_stamp":"2023-11-25T05:26:31.157416+0000","last_clean_scrub_stamp":"2023-11-25T05:26:31.157416+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:20:18.589017+0000","scrub_duration":0.072406576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:24.156130+0000","last_change":"2023-11-25T05:26:24.156858+0000","last_active":"2023-11-25T05:26:24.156130+0000","last_peered":"2023-11-25T05:26:24.156130+0000","last_clean":"2023-11-25T05:26:24.156130+0000","last_became_active":"2023-11-25T05:05:49.941593+0000","last_became_peered":"2023-11-25T05:05:49.941593+0000","last_unstale":"2023-11-25T05:26:24.156130+0000","last_undegraded":"2023-11-25T05:26:24.156130+0000","last_fullsized":"2023-11-25T05:26:24.156130+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:24.154721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:24.154721+0000","last_clean_scrub_stamp":"2023-11-25T05:26:24.154721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T10:11:07.189026+0000","scrub_duration":0.037000584000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:30.074375+0000","last_change":"2023-11-25T05:26:30.075111+0000","last_active":"2023-11-25T05:26:30.074375+0000","last_peered":"2023-11-25T05:26:30.074375+0000","last_clean":"2023-11-25T05:26:30.074375+0000","last_became_active":"2023-11-25T05:06:26.500023+0000","last_became_peered":"2023-11-25T05:06:26.500023+0000","last_unstale":"2023-11-25T05:26:30.074375+0000","last_undegraded":"2023-11-25T05:26:30.074375+0000","last_fullsized":"2023-11-25T05:26:30.074375+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:30.072946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:30.072946+0000","last_clean_scrub_stamp":"2023-11-25T05:26:30.072946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T09:35:15.693042+0000","scrub_duration":0.060247930999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":877,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:30.118560+0000","last_change":"2023-11-25T05:26:30.118560+0000","last_active":"2023-11-25T05:26:30.118560+0000","last_peered":"2023-11-25T05:26:30.118560+0000","last_clean":"2023-11-25T05:26:30.118560+0000","last_became_active":"2023-11-25T05:05:48.093083+0000","last_became_peered":"2023-11-25T05:05:48.093083+0000","last_unstale":"2023-11-25T05:26:30.118560+0000","last_undegraded":"2023-11-25T05:26:30.118560+0000","last_fullsized":"2023-11-25T05:26:30.118560+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:30.116574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:30.116574+0000","last_clean_scrub_stamp":"2023-11-25T05:26:30.116574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T06:57:04.485129+0000","scrub_duration":0.033886922999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:35.168400+0000","last_change":"2023-11-25T05:26:35.168400+0000","last_active":"2023-11-25T05:26:35.168400+0000","last_peered":"2023-11-25T05:26:35.168400+0000","last_clean":"2023-11-25T05:26:35.168400+0000","last_became_active":"2023-11-25T05:05:55.006070+0000","last_became_peered":"2023-11-25T05:05:55.006070+0000","last_unstale":"2023-11-25T05:26:35.168400+0000","last_undegraded":"2023-11-25T05:26:35.168400+0000","last_fullsized":"2023-11-25T05:26:35.168400+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:35.166401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:35.166401+0000","last_clean_scrub_stamp":"2023-11-25T05:26:35.166401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T16:53:47.955364+0000","scrub_duration":0.045775001000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:34.143819+0000","last_change":"2023-11-25T05:26:34.144478+0000","last_active":"2023-11-25T05:26:34.143819+0000","last_peered":"2023-11-25T05:26:34.143819+0000","last_clean":"2023-11-25T05:26:34.143819+0000","last_became_active":"2023-11-25T05:05:42.327418+0000","last_became_peered":"2023-11-25T05:05:42.327418+0000","last_unstale":"2023-11-25T05:26:34.143819+0000","last_undegraded":"2023-11-25T05:26:34.143819+0000","last_fullsized":"2023-11-25T05:26:34.143819+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:34.138526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:34.138526+0000","last_clean_scrub_stamp":"2023-11-25T05:26:34.138526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T07:13:40.861506+0000","scrub_duration":0.030268010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:31.961952+0000","last_change":"2023-11-25T05:26:31.962675+0000","last_active":"2023-11-25T05:26:31.961952+0000","last_peered":"2023-11-25T05:26:31.961952+0000","last_clean":"2023-11-25T05:26:31.961952+0000","last_became_active":"2023-11-25T05:05:45.334024+0000","last_became_peered":"2023-11-25T05:05:45.334024+0000","last_unstale":"2023-11-25T05:26:31.961952+0000","last_undegraded":"2023-11-25T05:26:31.961952+0000","last_fullsized":"2023-11-25T05:26:31.961952+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:31.959134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:31.959134+0000","last_clean_scrub_stamp":"2023-11-25T05:26:31.959134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T09:57:27.882194+0000","scrub_duration":0.074298776999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:32.028935+0000","last_change":"2023-11-25T05:26:32.030103+0000","last_active":"2023-11-25T05:26:32.028935+0000","last_peered":"2023-11-25T05:26:32.028935+0000","last_clean":"2023-11-25T05:26:32.028935+0000","last_became_active":"2023-11-25T05:06:27.122861+0000","last_became_peered":"2023-11-25T05:06:27.122861+0000","last_unstale":"2023-11-25T05:26:32.028935+0000","last_undegraded":"2023-11-25T05:26:32.028935+0000","last_fullsized":"2023-11-25T05:26:32.028935+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:32.026758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:32.026758+0000","last_clean_scrub_stamp":"2023-11-25T05:26:32.026758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T12:34:00.052329+0000","scrub_duration":0.043541608000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":881,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-11-25T05:26:36.853733+0000","last_change":"2023-11-25T05:26:36.854435+0000","last_active":"2023-11-25T05:26:36.853733+0000","last_peered":"2023-11-25T05:26:36.853733+0000","last_clean":"2023-11-25T05:26:36.853733+0000","last_became_active":"2023-11-25T05:05:57.177615+0000","last_became_peered":"2023-11-25T05:05:57.177615+0000","last_unstale":"2023-11-25T05:26:36.853733+0000","last_undegraded":"2023-11-25T05:26:36.853733+0000","last_fullsized":"2023-11-25T05:26:36.853733+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-25T05:26:36.852342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-25T05:26:36.852342+0000","last_clean_scrub_stamp":"2023-11-25T05:26:36.852342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-26T13:33:40.863507+0000","scrub_duration":0.050814239999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":16,"seq":68719477009,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66664,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305176,"statfs":{"total":96636764160,"available":96568500224,"internally_reserved":0,"allocated":13598720,"data_stored":8282535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:27:03 2023","interfaces":[{"interface":"back","average":{"1min":12.356,"5min":21.532,"15min":23.523},"min":{"1min":7.0780000000000003,"5min":4.9020000000000001,"15min":4.9020000000000001},"max":{"1min":32.191000000000003,"5min":164.57300000000001,"15min":200.26599999999999},"last":11.282},{"interface":"front","average":{"1min":10.917999999999999,"5min":21.393999999999998,"15min":24.228000000000002},"min":{"1min":6.9279999999999999,"5min":5.6859999999999999,"15min":5.4400000000000004},"max":{"1min":22.808,"5min":238.51300000000001,"15min":347.452},"last":6.9279999999999999}]},{"osd":1,"last update":"Sat Nov 25 05:27:03 2023","interfaces":[{"interface":"back","average":{"1min":14.847,"5min":23.904,"15min":23.673999999999999},"min":{"1min":8.2140000000000004,"5min":7.4660000000000002,"15min":5.6589999999999998},"max":{"1min":31.425999999999998,"5min":287.27199999999999,"15min":287.27199999999999},"last":14.112},{"interface":"front","average":{"1min":14.173,"5min":23.82,"15min":22.622},"min":{"1min":8.4589999999999996,"5min":5.282,"15min":5.282},"max":{"1min":24.463000000000001,"5min":286.57400000000001,"15min":286.57400000000001},"last":8.4589999999999996}]},{"osd":3,"last update":"Sat Nov 25 05:27:03 2023","interfaces":[{"interface":"back","average":{"1min":9.593,"5min":18.837,"15min":18.530000000000001},"min":{"1min":5.7919999999999998,"5min":5.7919999999999998,"15min":5.7919999999999998},"max":{"1min":17.765000000000001,"5min":167.83799999999999,"15min":177.66},"last":9.2889999999999997},{"interface":"front","average":{"1min":15.228,"5min":25.652000000000001,"15min":27.812000000000001},"min":{"1min":9.8879999999999999,"5min":8.1829999999999998,"15min":7.609},"max":{"1min":25.221,"5min":156.17699999999999,"15min":273.99900000000002},"last":16.611000000000001}]},{"osd":4,"last update":"Sat Nov 25 05:27:03 2023","interfaces":[{"interface":"back","average":{"1min":12.175000000000001,"5min":24.931999999999999,"15min":25.379000000000001},"min":{"1min":7.1130000000000004,"5min":6.9109999999999996,"15min":6.9109999999999996},"max":{"1min":22.989999999999998,"5min":275.66300000000001,"15min":349.90300000000002},"last":15.01},{"interface":"front","average":{"1min":17.506,"5min":23.298999999999999,"15min":21.262},"min":{"1min":7.8719999999999999,"5min":7.1589999999999998,"15min":6.6760000000000002},"max":{"1min":68.762,"5min":285.12700000000001,"15min":285.12700000000001},"last":10.427}]},{"osd":5,"last update":"Sat Nov 25 05:27:03 2023","interfaces":[{"interface":"back","average":{"1min":14.929,"5min":22.803000000000001,"15min":25.196999999999999},"min":{"1min":9.0350000000000001,"5min":9.0350000000000001,"15min":8.0619999999999994},"max":{"1min":22.149000000000001,"5min":138.34899999999999,"15min":303.452},"last":11.984999999999999},{"interface":"front","average":{"1min":19.012,"5min":28.675999999999998,"15min":30.417000000000002},"min":{"1min":15.396000000000001,"5min":14.381,"15min":11.539999999999999},"max":{"1min":32.807000000000002,"5min":146.41300000000001,"15min":304.15899999999999},"last":19.227}]},{"osd":6,"last update":"Sat Nov 25 05:27:03 2023","interfaces":[{"interface":"back","average":{"1min":18.670000000000002,"5min":27.123999999999999,"15min":29.283999999999999},"min":{"1min":12.871,"5min":11.292999999999999,"15min":11.292999999999999},"max":{"1min":34.337000000000003,"5min":153.01400000000001,"15min":213.11000000000001},"last":18.584},{"interface":"front","average":{"1min":20.515999999999998,"5min":29.619,"15min":32.771000000000001},"min":{"1min":16.013999999999999,"5min":12.462999999999999,"15min":12.462999999999999},"max":{"1min":40.323,"5min":144.51400000000001,"15min":288.30000000000001},"last":20.428999999999998}]},{"osd":7,"last update":"Sat Nov 25 05:27:03 2023","interfaces":[{"interface":"back","average":{"1min":16.876000000000001,"5min":24.248999999999999,"15min":26.454999999999998},"min":{"1min":10.744,"5min":9.1869999999999994,"15min":6.375},"max":{"1min":29.309000000000001,"5min":139.095,"15min":350.81599999999997},"last":15.815},{"interface":"front","average":{"1min":12.779,"5min":19.934999999999999,"15min":22.109999999999999},"min":{"1min":9.1669999999999998,"5min":7.5419999999999998,"15min":7.1669999999999998},"max":{"1min":17.677,"5min":104.87,"15min":210.55500000000001},"last":12.75}]}]},{"osd":1,"up_from":16,"seq":68719477009,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67056,"kb_used_data":13672,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94304784,"statfs":{"total":96636764160,"available":96568098816,"internally_reserved":0,"allocated":14000128,"data_stored":8680375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:27:04 2023","interfaces":[{"interface":"back","average":{"1min":11.944000000000001,"5min":17.957999999999998,"15min":20.824000000000002},"min":{"1min":5.5170000000000003,"5min":5.5170000000000003,"15min":5.1600000000000001},"max":{"1min":35.331000000000003,"5min":163.62,"15min":349.75799999999998},"last":7.641},{"interface":"front","average":{"1min":16.042000000000002,"5min":25.968,"15min":25.611999999999998},"min":{"1min":8.0980000000000008,"5min":8.0980000000000008,"15min":6.5309999999999997},"max":{"1min":31.024000000000001,"5min":240.31200000000001,"15min":318.02600000000001},"last":12.183999999999999}]},{"osd":2,"last update":"Sat Nov 25 05:27:04 2023","interfaces":[{"interface":"back","average":{"1min":12.273999999999999,"5min":18.59,"15min":21.745000000000001},"min":{"1min":6.319,"5min":5.1429999999999998,"15min":4.9619999999999997},"max":{"1min":27.486999999999998,"5min":161.10900000000001,"15min":353.286},"last":18.783999999999999},{"interface":"front","average":{"1min":13.661,"5min":19.745000000000001,"15min":18.881},"min":{"1min":6.9219999999999997,"5min":6.3280000000000003,"15min":5.7149999999999999},"max":{"1min":39.329000000000001,"5min":169.84100000000001,"15min":247.99199999999999},"last":8.516}]},{"osd":3,"last update":"Sat Nov 25 05:27:04 2023","interfaces":[{"interface":"back","average":{"1min":18.998999999999999,"5min":27.478999999999999,"15min":23.969000000000001},"min":{"1min":7.8810000000000002,"5min":6.665,"15min":6.5899999999999999},"max":{"1min":86.290999999999997,"5min":244.58199999999999,"15min":440.05599999999998},"last":16.045999999999999},{"interface":"front","average":{"1min":17.611999999999998,"5min":28.497,"15min":26.866},"min":{"1min":9.2029999999999994,"5min":8.4369999999999994,"15min":5.9889999999999999},"max":{"1min":34.197000000000003,"5min":242.29400000000001,"15min":383.39800000000002},"last":9.2029999999999994}]},{"osd":4,"last update":"Sat Nov 25 05:27:04 2023","interfaces":[{"interface":"back","average":{"1min":14.183,"5min":16.427,"15min":20.268999999999998},"min":{"1min":8.6280000000000001,"5min":6.7830000000000004,"15min":6.7830000000000004},"max":{"1min":34.058,"5min":78.707999999999998,"15min":360.161},"last":14.587999999999999},{"interface":"front","average":{"1min":16.808,"5min":22.058,"15min":22.934000000000001},"min":{"1min":8.0570000000000004,"5min":8.0570000000000004,"15min":7.4409999999999998},"max":{"1min":37.947000000000003,"5min":114.33199999999999,"15min":364.39800000000002},"last":10.832000000000001}]},{"osd":5,"last update":"Sat Nov 25 05:27:04 2023","interfaces":[{"interface":"back","average":{"1min":27.882999999999999,"5min":29.228999999999999,"15min":29.207999999999998},"min":{"1min":11.548999999999999,"5min":9.6669999999999998,"15min":7.4359999999999999},"max":{"1min":121.694,"5min":157.357,"15min":282.63900000000001},"last":13.082000000000001},{"interface":"front","average":{"1min":24.463000000000001,"5min":24.690000000000001,"15min":26.465},"min":{"1min":10.016999999999999,"5min":7.6630000000000003,"15min":7.1529999999999996},"max":{"1min":126.55,"5min":163.33600000000001,"15min":382.01400000000001},"last":10.016999999999999}]},{"osd":6,"last update":"Sat Nov 25 05:26:23 2023","interfaces":[{"interface":"back","average":{"1min":22.631,"5min":23.855,"15min":27.713999999999999},"min":{"1min":9.2360000000000007,"5min":7.5209999999999999,"15min":6.8230000000000004},"max":{"1min":71.406999999999996,"5min":187.62100000000001,"15min":493.18799999999999},"last":13.941000000000001},{"interface":"front","average":{"1min":26.407,"5min":26.518999999999998,"15min":30.838000000000001},"min":{"1min":13.795,"5min":9.9260000000000002,"15min":9.423},"max":{"1min":75.549999999999997,"5min":131.99600000000001,"15min":387.71199999999999},"last":15.266}]},{"osd":7,"last update":"Sat Nov 25 05:27:04 2023","interfaces":[{"interface":"back","average":{"1min":18.036000000000001,"5min":21.346,"15min":23.986999999999998},"min":{"1min":8.2309999999999999,"5min":7.6459999999999999,"15min":7.0279999999999996},"max":{"1min":38.832000000000001,"5min":157.755,"15min":272.03699999999998},"last":19.504999999999999},{"interface":"front","average":{"1min":21.097999999999999,"5min":25.620000000000001,"15min":27.242000000000001},"min":{"1min":11.481999999999999,"5min":11.481999999999999,"15min":8.3889999999999993},"max":{"1min":35.962000000000003,"5min":126.80200000000001,"15min":297.976},"last":11.481999999999999}]}]},{"osd":3,"up_from":16,"seq":68719477010,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84712,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94287128,"statfs":{"total":96636764160,"available":96550019072,"internally_reserved":0,"allocated":13598720,"data_stored":8282535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:27:06 2023","interfaces":[{"interface":"back","average":{"1min":11.567,"5min":16.620999999999999,"15min":20.361999999999998},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3090000000000002},"max":{"1min":36.960999999999999,"5min":124.46299999999999,"15min":394.84199999999998},"last":21.474},{"interface":"front","average":{"1min":12.24,"5min":17.277000000000001,"15min":20.466999999999999},"min":{"1min":4.54,"5min":4.54,"15min":4.54},"max":{"1min":37.802,"5min":119.611,"15min":379.51900000000001},"last":10.744999999999999}]},{"osd":1,"last update":"Sat Nov 25 05:26:14 2023","interfaces":[{"interface":"back","average":{"1min":21.181999999999999,"5min":20.585000000000001,"15min":28.129999999999999},"min":{"1min":7.6379999999999999,"5min":7.1779999999999999,"15min":5.4080000000000004},"max":{"1min":75.337999999999994,"5min":132.477,"15min":674.39599999999996},"last":14.862},{"interface":"front","average":{"1min":18.774000000000001,"5min":24.016999999999999,"15min":26.608000000000001},"min":{"1min":6.1829999999999998,"5min":6.1829999999999998,"15min":6.1829999999999998},"max":{"1min":62.155999999999999,"5min":253.03100000000001,"15min":399.98500000000001},"last":12.769}]},{"osd":2,"last update":"Sat Nov 25 05:26:23 2023","interfaces":[{"interface":"back","average":{"1min":25.303000000000001,"5min":17.934000000000001,"15min":29.198},"min":{"1min":8.8580000000000005,"5min":7.2649999999999997,"15min":7.0620000000000003},"max":{"1min":84.911000000000001,"5min":84.911000000000001,"15min":673.12400000000002},"last":16.196000000000002},{"interface":"front","average":{"1min":26.186,"5min":23.507000000000001,"15min":29.689},"min":{"1min":9.1300000000000008,"5min":7.351,"15min":7.351},"max":{"1min":79.689999999999998,"5min":227.36199999999999,"15min":462.27699999999999},"last":13.427}]},{"osd":4,"last update":"Sat Nov 25 05:27:06 2023","interfaces":[{"interface":"back","average":{"1min":15.779999999999999,"5min":20.934999999999999,"15min":28.658000000000001},"min":{"1min":10.723000000000001,"5min":8.8360000000000003,"15min":8.6440000000000001},"max":{"1min":22.710000000000001,"5min":141.38399999999999,"15min":676.21299999999997},"last":12.042},{"interface":"front","average":{"1min":11.144,"5min":19.405000000000001,"15min":21.939},"min":{"1min":8.0999999999999996,"5min":6.641,"15min":6.1559999999999997},"max":{"1min":15.348000000000001,"5min":313.18299999999999,"15min":313.18299999999999},"last":10.208}]},{"osd":5,"last update":"Sat Nov 25 05:27:06 2023","interfaces":[{"interface":"back","average":{"1min":16.818000000000001,"5min":22.391999999999999,"15min":31.172000000000001},"min":{"1min":11.536,"5min":10.542,"15min":9.3360000000000003},"max":{"1min":20.728000000000002,"5min":194.51400000000001,"15min":675.37800000000004},"last":15.542999999999999},{"interface":"front","average":{"1min":17.181999999999999,"5min":21.416,"15min":26.943999999999999},"min":{"1min":11.007999999999999,"5min":7.9429999999999996,"15min":7.5190000000000001},"max":{"1min":29.794,"5min":131.768,"15min":314.40800000000002},"last":20.309999999999999}]},{"osd":6,"last update":"Sat Nov 25 05:27:06 2023","interfaces":[{"interface":"back","average":{"1min":13.16,"5min":18.707999999999998,"15min":23.375},"min":{"1min":7.8319999999999999,"5min":7.8319999999999999,"15min":7.0190000000000001},"max":{"1min":18.114000000000001,"5min":136.11099999999999,"15min":312.20699999999999},"last":17.829000000000001},{"interface":"front","average":{"1min":17.280999999999999,"5min":21.937000000000001,"15min":32.834000000000003},"min":{"1min":10.135,"5min":10.135,"15min":10.029},"max":{"1min":21.332000000000001,"5min":156.55500000000001,"15min":677.03499999999997},"last":16.870000000000001}]},{"osd":7,"last update":"Sat Nov 25 05:27:06 2023","interfaces":[{"interface":"back","average":{"1min":15.456,"5min":23.504999999999999,"15min":26.079999999999998},"min":{"1min":9.4060000000000006,"5min":9.4030000000000005,"15min":7.6420000000000003},"max":{"1min":19.818999999999999,"5min":315.61500000000001,"15min":315.61500000000001},"last":19.628},{"interface":"front","average":{"1min":16.672999999999998,"5min":23.917999999999999,"15min":29.794},"min":{"1min":10.839,"5min":10.535,"15min":10.535},"max":{"1min":25.646000000000001,"5min":145.86099999999999,"15min":320.38400000000001},"last":22.838000000000001}]}]},{"osd":0,"up_from":16,"seq":68719477011,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66664,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305176,"statfs":{"total":96636764160,"available":96568500224,"internally_reserved":0,"allocated":13598720,"data_stored":8282535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Nov 25 05:27:03 2023","interfaces":[{"interface":"back","average":{"1min":12.127000000000001,"5min":13.65,"15min":17.045999999999999},"min":{"1min":6.2649999999999997,"5min":4.6100000000000003,"15min":4.6100000000000003},"max":{"1min":28.975000000000001,"5min":82.027000000000001,"15min":209.29900000000001},"last":8.2100000000000009},{"interface":"front","average":{"1min":12.676,"5min":14.794,"15min":18.960000000000001},"min":{"1min":6.673,"5min":4.5869999999999997,"15min":4.4420000000000002},"max":{"1min":31.597999999999999,"5min":119.73,"15min":401.36000000000001},"last":11.673999999999999}]},{"osd":2,"last update":"Sat Nov 25 05:27:03 2023","interfaces":[{"interface":"back","average":{"1min":14.753,"5min":17.661999999999999,"15min":21.332000000000001},"min":{"1min":8.1760000000000002,"5min":6.5380000000000003,"15min":6.1260000000000003},"max":{"1min":42.573999999999998,"5min":94.311999999999998,"15min":460.572},"last":13.153},{"interface":"front","average":{"1min":15.582000000000001,"5min":17.567,"15min":21.34},"min":{"1min":9.1349999999999998,"5min":6.5279999999999996,"15min":6.3090000000000002},"max":{"1min":37.648000000000003,"5min":76.372,"15min":405.94099999999997},"last":9.0690000000000008}]},{"osd":3,"last update":"Sat Nov 25 05:27:03 2023","interfaces":[{"interface":"back","average":{"1min":16.599,"5min":21.966000000000001,"15min":23.370000000000001},"min":{"1min":7.6260000000000003,"5min":7.2999999999999998,"15min":6.8540000000000001},"max":{"1min":57.186999999999998,"5min":157.65299999999999,"15min":267.74599999999998},"last":12.462999999999999},{"interface":"front","average":{"1min":15.602,"5min":20.905000000000001,"15min":22.321999999999999},"min":{"1min":8.3569999999999993,"5min":7.1269999999999998,"15min":6.4080000000000004},"max":{"1min":37.729999999999997,"5min":128.14500000000001,"15min":266.81900000000002},"last":13.901999999999999}]},{"osd":4,"last update":"Sat Nov 25 05:27:03 2023","interfaces":[{"interface":"back","average":{"1min":14.571,"5min":17.417000000000002,"15min":19.478999999999999},"min":{"1min":8.5459999999999994,"5min":8.3940000000000001,"15min":6.2750000000000004},"max":{"1min":30.289000000000001,"5min":72.540999999999997,"15min":147.86099999999999},"last":17.285},{"interface":"front","average":{"1min":16.044,"5min":20.577000000000002,"15min":21.468},"min":{"1min":8.0559999999999992,"5min":7.9219999999999997,"15min":6.8250000000000002},"max":{"1min":37.064999999999998,"5min":110.93300000000001,"15min":148.96199999999999},"last":18.032}]},{"osd":5,"last update":"Sat Nov 25 05:27:03 2023","interfaces":[{"interface":"back","average":{"1min":23.038,"5min":21.545000000000002,"15min":22.777999999999999},"min":{"1min":9.5039999999999996,"5min":6.9829999999999997,"15min":6.9829999999999997},"max":{"1min":136,"5min":136,"15min":184.80000000000001},"last":20.248000000000001},{"interface":"front","average":{"1min":30.260000000000002,"5min":23.664000000000001,"15min":25.327999999999999},"min":{"1min":9.4809999999999999,"5min":9.4809999999999999,"15min":7.3849999999999998},"max":{"1min":227.119,"5min":227.119,"15min":395.65100000000001},"last":16.396000000000001}]},{"osd":6,"last update":"Sat Nov 25 05:27:03 2023","interfaces":[{"interface":"back","average":{"1min":17.449999999999999,"5min":21.818999999999999,"15min":22.949999999999999},"min":{"1min":10.298999999999999,"5min":8.4809999999999999,"15min":8.4809999999999999},"max":{"1min":44.402999999999999,"5min":86.356999999999999,"15min":140.16499999999999},"last":18.838999999999999},{"interface":"front","average":{"1min":19.904,"5min":24.492999999999999,"15min":26.016999999999999},"min":{"1min":12.355,"5min":9.7759999999999998,"15min":9.7759999999999998},"max":{"1min":54.859999999999999,"5min":90.045000000000002,"15min":138.46799999999999},"last":20.965}]},{"osd":7,"last update":"Sat Nov 25 05:27:03 2023","interfaces":[{"interface":"back","average":{"1min":16.838000000000001,"5min":17.978000000000002,"15min":19.914999999999999},"min":{"1min":10.441000000000001,"5min":8.7599999999999998,"15min":7.1079999999999997},"max":{"1min":33.460000000000001,"5min":84.399000000000001,"15min":140.595},"last":10.212},{"interface":"front","average":{"1min":20.798999999999999,"5min":21.978999999999999,"15min":24.629000000000001},"min":{"1min":8.984,"5min":8.984,"15min":8.984},"max":{"1min":42.442999999999998,"5min":83.198999999999998,"15min":339.298},"last":14.709}]}]},{"osd":6,"up_from":15,"seq":64424509714,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84712,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94287128,"statfs":{"total":96636764160,"available":96550019072,"internally_reserved":0,"allocated":13598720,"data_stored":8282535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:27:02 2023","interfaces":[{"interface":"back","average":{"1min":12.234999999999999,"5min":15.039,"15min":18.609000000000002},"min":{"1min":5.5090000000000003,"5min":4.7489999999999997,"15min":4.4409999999999998},"max":{"1min":25.263000000000002,"5min":117.05500000000001,"15min":183.05500000000001},"last":8.5890000000000004},{"interface":"front","average":{"1min":8.0269999999999992,"5min":13.090999999999999,"15min":19.286000000000001},"min":{"1min":3.8809999999999998,"5min":3.8809999999999998,"15min":3.8809999999999998},"max":{"1min":31.957999999999998,"5min":117.795,"15min":381.43900000000002},"last":5.7389999999999999}]},{"osd":1,"last update":"Sat Nov 25 05:27:02 2023","interfaces":[{"interface":"back","average":{"1min":13.603999999999999,"5min":18.550000000000001,"15min":25.129000000000001},"min":{"1min":6.7809999999999997,"5min":6.2839999999999998,"15min":6.165},"max":{"1min":17.966000000000001,"5min":165.59800000000001,"15min":310.423},"last":11.396000000000001},{"interface":"front","average":{"1min":10.124000000000001,"5min":16.154,"15min":25.102},"min":{"1min":5.9429999999999996,"5min":5.9429999999999996,"15min":5.1310000000000002},"max":{"1min":18.731000000000002,"5min":190.029,"15min":548.03599999999994},"last":9.7080000000000002}]},{"osd":2,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":28.033999999999999,"5min":22.832000000000001,"15min":27.013999999999999},"min":{"1min":7.9550000000000001,"5min":6.9690000000000003,"15min":6.9690000000000003},"max":{"1min":122.754,"5min":251.524,"15min":353.80399999999997},"last":14.204000000000001},{"interface":"front","average":{"1min":26.027000000000001,"5min":27.611999999999998,"15min":30.841999999999999},"min":{"1min":10.65,"5min":10.65,"15min":10.617000000000001},"max":{"1min":120.961,"5min":249.97399999999999,"15min":304.49900000000002},"last":12.851000000000001}]},{"osd":3,"last update":"Sat Nov 25 05:27:02 2023","interfaces":[{"interface":"back","average":{"1min":24.056999999999999,"5min":28.404,"15min":32.021999999999998},"min":{"1min":12.278,"5min":10.164,"15min":7.5419999999999998},"max":{"1min":156.45099999999999,"5min":259.89299999999997,"15min":295.82799999999997},"last":12.278},{"interface":"front","average":{"1min":20.120000000000001,"5min":24.481999999999999,"15min":26.169},"min":{"1min":6.9550000000000001,"5min":6.9199999999999999,"15min":6.0209999999999999},"max":{"1min":181.51599999999999,"5min":240.71299999999999,"15min":266.40199999999999},"last":13.484}]},{"osd":4,"last update":"Sat Nov 25 05:27:02 2023","interfaces":[{"interface":"back","average":{"1min":17.206,"5min":26.606000000000002,"15min":30.922000000000001},"min":{"1min":11.523999999999999,"5min":10.000999999999999,"15min":10.000999999999999},"max":{"1min":25.856999999999999,"5min":237.58500000000001,"15min":330.86900000000003},"last":17.222000000000001},{"interface":"front","average":{"1min":13.148,"5min":21.387,"15min":24.532},"min":{"1min":6.819,"5min":6.819,"15min":6.5919999999999996},"max":{"1min":29.606000000000002,"5min":200.5,"15min":288.13099999999997},"last":14.946}]},{"osd":5,"last update":"Sat Nov 25 05:27:02 2023","interfaces":[{"interface":"back","average":{"1min":9.1120000000000001,"5min":13.99,"15min":21.489999999999998},"min":{"1min":6.0220000000000002,"5min":5.6769999999999996,"15min":5.5590000000000002},"max":{"1min":12.818,"5min":121.31399999999999,"15min":281.13999999999999},"last":10.079000000000001},{"interface":"front","average":{"1min":18.021000000000001,"5min":23.282,"15min":31.573},"min":{"1min":13.314,"5min":8.0069999999999997,"15min":8.0069999999999997},"max":{"1min":23.146999999999998,"5min":133.44999999999999,"15min":311.09199999999998},"last":18.942}]},{"osd":7,"last update":"Sat Nov 25 05:27:02 2023","interfaces":[{"interface":"back","average":{"1min":18.966999999999999,"5min":26.715,"15min":34.369999999999997},"min":{"1min":14.119,"5min":12.319000000000001,"15min":9.2210000000000001},"max":{"1min":28.085999999999999,"5min":164.54599999999999,"15min":367.24900000000002},"last":18.184999999999999},{"interface":"front","average":{"1min":14.292,"5min":20.181000000000001,"15min":27.524999999999999},"min":{"1min":9.8339999999999996,"5min":8.3469999999999995,"15min":7.7199999999999998},"max":{"1min":30.277000000000001,"5min":161.37899999999999,"15min":325.54199999999997},"last":16.492999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509716,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70828,"kb_used_data":13284,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94301012,"statfs":{"total":96636764160,"available":96564236288,"internally_reserved":0,"allocated":13602816,"data_stored":8282540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:26 2023","interfaces":[{"interface":"back","average":{"1min":18.672999999999998,"5min":18.614000000000001,"15min":19.898},"min":{"1min":4.8550000000000004,"5min":4.3170000000000002,"15min":3.8279999999999998},"max":{"1min":124.38200000000001,"5min":150.73599999999999,"15min":219.172},"last":7.1120000000000001},{"interface":"front","average":{"1min":19.463999999999999,"5min":17.724,"15min":18.888000000000002},"min":{"1min":4.9299999999999997,"5min":4.5369999999999999,"15min":4.1289999999999996},"max":{"1min":204.953,"5min":216.86600000000001,"15min":216.86600000000001},"last":5.9199999999999999}]},{"osd":1,"last update":"Sat Nov 25 05:26:26 2023","interfaces":[{"interface":"back","average":{"1min":15.097,"5min":18.919,"15min":18.690000000000001},"min":{"1min":5.79,"5min":5.2149999999999999,"15min":4.9749999999999996},"max":{"1min":89.542000000000002,"5min":205.36500000000001,"15min":205.36500000000001},"last":7.7640000000000002},{"interface":"front","average":{"1min":25.373000000000001,"5min":21.013999999999999,"15min":20.709},"min":{"1min":8.6920000000000002,"5min":6.5759999999999996,"15min":6.5759999999999996},"max":{"1min":196.667,"5min":196.667,"15min":196.667},"last":14.926}]},{"osd":2,"last update":"Sat Nov 25 05:26:26 2023","interfaces":[{"interface":"back","average":{"1min":28.908000000000001,"5min":21.404,"15min":23.119},"min":{"1min":8.8520000000000003,"5min":6.9320000000000004,"15min":6.9320000000000004},"max":{"1min":277.51299999999998,"5min":277.51299999999998,"15min":277.51299999999998},"last":15.57},{"interface":"front","average":{"1min":31.375,"5min":24.786999999999999,"15min":26.934000000000001},"min":{"1min":14.608000000000001,"5min":10.664,"15min":8.5899999999999999},"max":{"1min":256.63,"5min":256.63,"15min":312.05200000000002},"last":16.585000000000001}]},{"osd":3,"last update":"Sat Nov 25 05:26:26 2023","interfaces":[{"interface":"back","average":{"1min":25.312000000000001,"5min":23.844000000000001,"15min":26.222999999999999},"min":{"1min":6.8760000000000003,"5min":6.8760000000000003,"15min":6.133},"max":{"1min":210.71299999999999,"5min":210.71299999999999,"15min":359.23700000000002},"last":10.919},{"interface":"front","average":{"1min":21.283999999999999,"5min":24.105,"15min":23.407},"min":{"1min":6.1719999999999997,"5min":6.1719999999999997,"15min":6.1719999999999997},"max":{"1min":137.21199999999999,"5min":196.55000000000001,"15min":381.53100000000001},"last":11.67}]},{"osd":5,"last update":"Sat Nov 25 05:26:58 2023","interfaces":[{"interface":"back","average":{"1min":21.855,"5min":22.533999999999999,"15min":26.440999999999999},"min":{"1min":7.9130000000000003,"5min":5.875,"15min":5.875},"max":{"1min":187.58699999999999,"5min":195.851,"15min":360.387},"last":13.872999999999999},{"interface":"front","average":{"1min":26.574000000000002,"5min":25.613,"15min":29.831},"min":{"1min":7.9539999999999997,"5min":7.9539999999999997,"15min":7.9539999999999997},"max":{"1min":213.98099999999999,"5min":213.98099999999999,"15min":274.38200000000001},"last":19.367000000000001}]},{"osd":6,"last update":"Sat Nov 25 05:26:58 2023","interfaces":[{"interface":"back","average":{"1min":29.309999999999999,"5min":26.004999999999999,"15min":27.07},"min":{"1min":13.335000000000001,"5min":10.093,"15min":9.8409999999999993},"max":{"1min":205.81899999999999,"5min":205.81899999999999,"15min":205.81899999999999},"last":20.495000000000001},{"interface":"front","average":{"1min":26.852,"5min":24.879999999999999,"15min":23.379000000000001},"min":{"1min":9.7159999999999993,"5min":7.2380000000000004,"15min":5.6390000000000002},"max":{"1min":211.68799999999999,"5min":212.25399999999999,"15min":219.26499999999999},"last":17.239999999999998}]},{"osd":7,"last update":"Sat Nov 25 05:26:58 2023","interfaces":[{"interface":"back","average":{"1min":26.364999999999998,"5min":24.946999999999999,"15min":25.577999999999999},"min":{"1min":10.627000000000001,"5min":7.0369999999999999,"15min":7.0369999999999999},"max":{"1min":195.61699999999999,"5min":202.363,"15min":215.512},"last":18.518999999999998},{"interface":"front","average":{"1min":22.905000000000001,"5min":22.131,"15min":24.556999999999999},"min":{"1min":6.9710000000000001,"5min":6.0350000000000001,"15min":6.0350000000000001},"max":{"1min":226.84200000000001,"5min":226.84200000000001,"15min":248.83199999999999},"last":13.202}]}]},{"osd":7,"up_from":13,"seq":55834575122,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66668,"kb_used_data":13284,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94305172,"statfs":{"total":96636764160,"available":96568496128,"internally_reserved":0,"allocated":13602816,"data_stored":8282540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":11.608000000000001,"5min":13.819000000000001,"15min":17.303999999999998},"min":{"1min":4.6150000000000002,"5min":4.3579999999999997,"15min":4.3579999999999997},"max":{"1min":23.535,"5min":97.569000000000003,"15min":222.50800000000001},"last":18.210999999999999},{"interface":"front","average":{"1min":9.8520000000000003,"5min":15.375999999999999,"15min":18.193999999999999},"min":{"1min":4.7450000000000001,"5min":4.6600000000000001,"15min":4.5579999999999998},"max":{"1min":25.248000000000001,"5min":218.03999999999999,"15min":218.03999999999999},"last":7.915}]},{"osd":1,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.643000000000001,"5min":13.715999999999999,"15min":17.489000000000001},"min":{"1min":5.4859999999999998,"5min":5.4859999999999998,"15min":5.2619999999999996},"max":{"1min":79.078999999999994,"5min":79.078999999999994,"15min":165.99000000000001},"last":12.968},{"interface":"front","average":{"1min":20.498000000000001,"5min":18.827000000000002,"15min":22.771999999999998},"min":{"1min":7.508,"5min":6.8140000000000001,"15min":6.7030000000000003},"max":{"1min":117.25,"5min":117.25,"15min":303.892},"last":9.2919999999999998}]},{"osd":2,"last update":"Sat Nov 25 05:26:18 2023","interfaces":[{"interface":"back","average":{"1min":12.206,"5min":18.088999999999999,"15min":20.989000000000001},"min":{"1min":6.3810000000000002,"5min":6.3810000000000002,"15min":6.3810000000000002},"max":{"1min":23.623999999999999,"5min":169.262,"15min":186.08600000000001},"last":16.268000000000001},{"interface":"front","average":{"1min":16.013000000000002,"5min":16.907,"15min":21.454000000000001},"min":{"1min":9.0570000000000004,"5min":7.508,"15min":6.4279999999999999},"max":{"1min":27.609999999999999,"5min":108.26600000000001,"15min":337.41899999999998},"last":11.102}]},{"osd":3,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":16.559999999999999,"5min":22.437999999999999,"15min":23.844999999999999},"min":{"1min":12.122999999999999,"5min":8.4469999999999992,"15min":7.5199999999999996},"max":{"1min":28.463000000000001,"5min":152.47399999999999,"15min":238.941},"last":17.16},{"interface":"front","average":{"1min":18.123000000000001,"5min":21.559999999999999,"15min":26.684999999999999},"min":{"1min":12.221,"5min":6.6929999999999996,"15min":5.5949999999999998},"max":{"1min":29.536000000000001,"5min":150.208,"15min":1292.413},"last":19.143000000000001}]},{"osd":4,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.506,"5min":23.238,"15min":25.268000000000001},"min":{"1min":8.8109999999999999,"5min":8.8109999999999999,"15min":7.9340000000000002},"max":{"1min":21.449999999999999,"5min":120.414,"15min":328.31400000000002},"last":10.207000000000001},{"interface":"front","average":{"1min":27.879000000000001,"5min":21.405000000000001,"15min":23.079999999999998},"min":{"1min":10.739000000000001,"5min":7.1849999999999996,"15min":6.4320000000000004},"max":{"1min":192.684,"5min":192.684,"15min":302.97199999999998},"last":20.010000000000002}]},{"osd":5,"last update":"Sat Nov 25 05:27:04 2023","interfaces":[{"interface":"back","average":{"1min":15.728999999999999,"5min":19.834,"15min":23.722000000000001},"min":{"1min":10.113,"5min":8.4890000000000008,"15min":7.7619999999999996},"max":{"1min":30.449000000000002,"5min":102.81999999999999,"15min":217.023},"last":12.287000000000001},{"interface":"front","average":{"1min":19.771999999999998,"5min":25.603000000000002,"15min":28.457000000000001},"min":{"1min":12.109,"5min":11.694000000000001,"15min":7.1130000000000004},"max":{"1min":32.600000000000001,"5min":110.935,"15min":318.11599999999999},"last":21.128}]},{"osd":6,"last update":"Sat Nov 25 05:26:12 2023","interfaces":[{"interface":"back","average":{"1min":14.75,"5min":20.798999999999999,"15min":22.933},"min":{"1min":7.069,"5min":6.2380000000000004,"15min":6.2380000000000004},"max":{"1min":100.828,"5min":188.15199999999999,"15min":282.28100000000001},"last":14.173999999999999},{"interface":"front","average":{"1min":18.027999999999999,"5min":23.329000000000001,"15min":25.382000000000001},"min":{"1min":8.0459999999999994,"5min":8.0459999999999994,"15min":8.0459999999999994},"max":{"1min":100.143,"5min":187.09899999999999,"15min":215.00700000000001},"last":14.890000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89264,"kb_used_data":13672,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94282576,"statfs":{"total":96636764160,"available":96545357824,"internally_reserved":0,"allocated":14000128,"data_stored":8680375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":17.623000000000001,"5min":14.541,"15min":18.605},"min":{"1min":4.5910000000000002,"5min":4.202,"15min":4.202},"max":{"1min":133.92599999999999,"5min":167.83600000000001,"15min":471.81},"last":7.8819999999999997},{"interface":"front","average":{"1min":14.865,"5min":15.755000000000001,"15min":20.606999999999999},"min":{"1min":6.2699999999999996,"5min":5.4279999999999999,"15min":4.5529999999999999},"max":{"1min":46.905000000000001,"5min":197.834,"15min":317.45699999999999},"last":9.0749999999999993}]},{"osd":1,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":27.741,"5min":18.356000000000002,"15min":21.565000000000001},"min":{"1min":6.3470000000000004,"5min":5.4189999999999996,"15min":5.0709999999999997},"max":{"1min":140.28999999999999,"5min":140.28999999999999,"15min":263.589},"last":18.838000000000001},{"interface":"front","average":{"1min":30.024000000000001,"5min":17.751000000000001,"15min":23.059999999999999},"min":{"1min":6.351,"5min":5.1459999999999999,"15min":5.1459999999999999},"max":{"1min":150.196,"5min":150.196,"15min":238.26499999999999},"last":9.8900000000000006}]},{"osd":2,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":19.062999999999999,"5min":19.102,"15min":23.210000000000001},"min":{"1min":7.1890000000000001,"5min":6.2240000000000002,"15min":6.2240000000000002},"max":{"1min":82.295000000000002,"5min":182.477,"15min":416.83199999999999},"last":11.343999999999999},{"interface":"front","average":{"1min":28.707000000000001,"5min":25.138000000000002,"15min":25.091999999999999},"min":{"1min":12.006,"5min":9.4000000000000004,"15min":7.0880000000000001},"max":{"1min":179.749,"5min":190.517,"15min":207.28700000000001},"last":13.589}]},{"osd":3,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":31.044,"5min":24.114000000000001,"15min":29.751999999999999},"min":{"1min":10.327,"5min":10.097,"15min":6.6079999999999997},"max":{"1min":166.59700000000001,"5min":166.59700000000001,"15min":443.10300000000001},"last":12.035},{"interface":"front","average":{"1min":28.024999999999999,"5min":19.308,"15min":31.902000000000001},"min":{"1min":7.851,"5min":6.9800000000000004,"15min":6.2199999999999998},"max":{"1min":130.87299999999999,"5min":130.87299999999999,"15min":1666.6489999999999},"last":12.869}]},{"osd":4,"last update":"Sat Nov 25 05:26:07 2023","interfaces":[{"interface":"back","average":{"1min":30.039000000000001,"5min":25.219999999999999,"15min":25.515999999999998},"min":{"1min":10.957000000000001,"5min":10.657,"15min":9.0280000000000005},"max":{"1min":173.041,"5min":173.041,"15min":173.041},"last":18.039000000000001},{"interface":"front","average":{"1min":17.225999999999999,"5min":17.971,"15min":21.449000000000002},"min":{"1min":6.5330000000000004,"5min":6.5330000000000004,"15min":6.5330000000000004},"max":{"1min":68.790000000000006,"5min":123.01300000000001,"15min":156.92500000000001},"last":10.666}]},{"osd":6,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":26.797999999999998,"5min":26.663,"15min":31.244},"min":{"1min":13.439,"5min":11.666,"15min":9.9979999999999993},"max":{"1min":100.47799999999999,"5min":229.86199999999999,"15min":229.86199999999999},"last":16.742999999999999},{"interface":"front","average":{"1min":31.841999999999999,"5min":24.629000000000001,"15min":27.375},"min":{"1min":8.6280000000000001,"5min":8.1159999999999997,"15min":7.4770000000000003},"max":{"1min":165.46000000000001,"5min":234.31100000000001,"15min":234.31100000000001},"last":15.98}]},{"osd":7,"last update":"Sat Nov 25 05:26:09 2023","interfaces":[{"interface":"back","average":{"1min":25.68,"5min":20.594999999999999,"15min":26.533999999999999},"min":{"1min":7.2679999999999998,"5min":6.2370000000000001,"15min":6.2370000000000001},"max":{"1min":67.870999999999995,"5min":195.94399999999999,"15min":195.94399999999999},"last":19.675000000000001},{"interface":"front","average":{"1min":23.308,"5min":25.666,"15min":26.741},"min":{"1min":11.151,"5min":11.151,"15min":7.3289999999999997},"max":{"1min":57.329000000000001,"5min":185.74700000000001,"15min":224.99600000000001},"last":15.288}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-25T05:27:09.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-11-25T05:27:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.351+0000 7f6d42f61640 10 monclient: get_monmap_and_config 2023-11-25T05:27:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.351+0000 7f6d42f61640 10 monclient: build_initial_monmap 2023-11-25T05:27:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.351+0000 7f6d42f61640 10 monclient: monmap: 2023-11-25T05:27:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:27:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:27:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:27:09.352726+0000 2023-11-25T05:27:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:27:09.352726+0000 2023-11-25T05:27:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:27:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:27:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-11-25T05:27:09.354 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c-legacy 2023-11-25T05:27:09.354 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-11-25T05:27:09.354 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-11-25T05:27:09.354 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.90:3300/0 mon.noname-c 2023-11-25T05:27:09.354 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:27:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.352+0000 7f6d42f61640 1 Processor -- start 2023-11-25T05:27:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.352+0000 7f6d42f61640 1 -- start start 2023-11-25T05:27:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.352+0000 7f6d42f61640 10 monclient: init 2023-11-25T05:27:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.353+0000 7f6d42f61640 10 monclient: _reopen_session rank -1 2023-11-25T05:27:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.353+0000 7f6d42f61640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-11-25T05:27:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.353+0000 7f6d42f61640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6d3c0a8930 0x7f6d3c0a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:27:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.353+0000 7f6d42f61640 10 monclient(hunting): picked mon.noname-a con 0x7f6d3c0a8930 addr v2:172.21.15.7:3300/0 2023-11-25T05:27:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.353+0000 7f6d42f61640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c13f170 0x7f6d3c13f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:27:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.353+0000 7f6d42f61640 10 monclient(hunting): picked mon.noname-b con 0x7f6d3c13f170 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:27:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.353+0000 7f6d42f61640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d3c145920 addr v1:172.21.15.90:6789/0 2023-11-25T05:27:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.353+0000 7f6d42f61640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d3c0aac60 con 0x7f6d3c145920 2023-11-25T05:27:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.353+0000 7f6d42f61640 10 monclient(hunting): start opening mon connection 2023-11-25T05:27:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.353+0000 7f6d42f61640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3c0aaed0 con 0x7f6d3c0a8930 2023-11-25T05:27:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.353+0000 7f6d42f61640 10 monclient(hunting): start opening mon connection 2023-11-25T05:27:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.353+0000 7f6d42f61640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c145400 con 0x7f6d3c13f170 2023-11-25T05:27:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.353+0000 7f6d42f61640 10 monclient(hunting): _renew_subs 2023-11-25T05:27:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.353+0000 7f6d42f61640 10 monclient(hunting): authenticate will time out at 2755.596168s 2023-11-25T05:27:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.353+0000 7f6d40cfd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6d3c0a8930 0x7f6d3c0a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:27:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.354+0000 7f6d414fe640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6d3c145920 0x7f6d3c145d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.7:39868/0 (socket says 172.21.15.7:39868) 2023-11-25T05:27:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.354+0000 7f6d414fe640 1 -- 172.21.15.7:0/1707940060 learned_addr learned my addr 172.21.15.7:0/1707940060 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-11-25T05:27:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.354+0000 7f6d3bfff640 1 --2- 172.21.15.7:0/1707940060 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c13f170 0x7f6d3c13f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:27:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.354+0000 7f6d40cfd640 10 monclient(hunting): get_auth_request con 0x7f6d3c0a8930 auth_method 0 2023-11-25T05:27:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.354+0000 7f6d40cfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:27:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.354+0000 7f6d40cfd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:27:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.354+0000 7f6d40cfd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:27:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.354+0000 7f6d3bfff640 10 monclient(hunting): get_auth_request con 0x7f6d3c13f170 auth_method 0 2023-11-25T05:27:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.354+0000 7f6d3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:27:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.354+0000 7f6d3bfff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:27:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.354+0000 7f6d3bfff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:27:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.355+0000 7f6d40cfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:27:09.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.355+0000 7f6d40cfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:27:09.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.355+0000 7f6d40cfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:27:09.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.355+0000 7f6d3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:27:09.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.355+0000 7f6d3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:27:09.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.355+0000 7f6d3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:27:09.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d3b7fe640 1 -- 172.21.15.7:0/1707940060 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565941324 0 0) 0x7f6d3c0aac60 con 0x7f6d3c145920 2023-11-25T05:27:09.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d3b7fe640 10 monclient(hunting): _negotiate 2023-11-25T05:27:09.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d3b7fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:27:09.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d3b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:27:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d3b7fe640 10 monclient(hunting): my global_id is 21322 2023-11-25T05:27:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d3b7fe640 1 -- 172.21.15.7:0/1707940060 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d28003520 con 0x7f6d3c145920 2023-11-25T05:27:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d3bfff640 10 monclient(hunting): handle_auth_done global_id 17459 payload 274 2023-11-25T05:27:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d3bfff640 1 -- 172.21.15.7:0/1707940060 >> v2:172.21.15.7:3300/0 conn(0x7f6d3c0a8930 msgr2=0x7f6d3c0a8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-25T05:27:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d3bfff640 1 --2- 172.21.15.7:0/1707940060 >> v2:172.21.15.7:3300/0 conn(0x7f6d3c0a8930 0x7f6d3c0a8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d3bfff640 1 -- 172.21.15.7:0/1707940060 >> v1:172.21.15.90:6789/0 conn(0x7f6d3c145920 legacy=0x7f6d3c145d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:27:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d3bfff640 10 monclient: _finish_hunting 0 2023-11-25T05:27:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d3bfff640 1 monclient: found mon.noname-b 2023-11-25T05:27:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:27:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d3bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-11-25T05:27:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d3bfff640 1 -- 172.21.15.7:0/1707940060 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3c0b3ad0 con 0x7f6d3c13f170 2023-11-25T05:27:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.356+0000 7f6d40cfd640 1 --2- 172.21.15.7:0/1707940060 >> v2:172.21.15.7:3300/0 conn(0x7f6d3c0a8930 0x7f6d3c0a8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-25T05:27:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.358+0000 7f6d3bfff640 1 --2- 172.21.15.7:0/1707940060 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c13f170 0x7f6d3c13f5d0 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7f6d240097a0 tx=0x7f6d2402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17480fe79af868f3 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:27:09.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d3b7fe640 1 -- 172.21.15.7:0/1707940060 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d24031830 con 0x7f6d3c13f170 2023-11-25T05:27:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:27:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d3b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:27:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d3b7fe640 10 monclient: dump: 2023-11-25T05:27:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:27:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:27:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:27:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:27:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:27:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:27:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:27:09.365 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:27:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d3b7fe640 10 monclient: _finish_auth 0 2023-11-25T05:27:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d3b7fe640 10 monclient: _check_auth_tickets 2023-11-25T05:27:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:27:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d3b7fe640 1 -- 172.21.15.7:0/1707940060 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d24031990 con 0x7f6d3c13f170 2023-11-25T05:27:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d3b7fe640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:27:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d3b7fe640 1 -- 172.21.15.7:0/1707940060 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d24031d20 con 0x7f6d3c13f170 2023-11-25T05:27:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:27:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d3b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-25T05:27:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d3b7fe640 10 monclient: dump: 2023-11-25T05:27:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:27:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:27:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:09.366 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:09.366 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:27:09.366 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:27:09.366 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:27:09.366 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:27:09.366 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:27:09.366 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:27:09.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d42f61640 5 monclient: authenticate success, global_id 17459 2023-11-25T05:27:09.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d42f61640 10 monclient: get_monmap_and_config success 2023-11-25T05:27:09.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.362+0000 7f6d42f61640 4 set_mon_vals no callback set 2023-11-25T05:27:09.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.363+0000 7f6d42f61640 10 monclient: shutdown 2023-11-25T05:27:09.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.363+0000 7f6d42f61640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:27:09.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.363+0000 7f6d42f61640 1 -- 172.21.15.7:0/1707940060 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c13f170 msgr2=0x7f6d3c13f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:27:09.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.363+0000 7f6d42f61640 1 --2- 172.21.15.7:0/1707940060 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c13f170 0x7f6d3c13f5d0 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7f6d240097a0 tx=0x7f6d2402b8d0 comp rx=0 tx=0).stop 2023-11-25T05:27:09.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.363+0000 7f6d42f61640 1 -- 172.21.15.7:0/1707940060 shutdown_connections 2023-11-25T05:27:09.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.363+0000 7f6d42f61640 1 --2- 172.21.15.7:0/1707940060 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c13f170 0x7f6d3c13f5d0 unknown :-1 s=CLOSED pgs=3079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:09.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.363+0000 7f6d42f61640 1 --2- 172.21.15.7:0/1707940060 >> v2:172.21.15.7:3300/0 conn(0x7f6d3c0a8930 0x7f6d3c0a8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:09.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.363+0000 7f6d42f61640 1 -- 172.21.15.7:0/1707940060 >> 172.21.15.7:0/1707940060 conn(0x7f6d3c0619c0 msgr2=0x7f6d3c0b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:27:09.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.363+0000 7f6d42f61640 1 -- 172.21.15.7:0/1707940060 shutdown_connections 2023-11-25T05:27:09.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.363+0000 7f6d42f61640 1 -- 172.21.15.7:0/1707940060 wait complete. 2023-11-25T05:27:09.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.364+0000 7f6d42f61640 10 monclient: build_initial_monmap 2023-11-25T05:27:09.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.364+0000 7f6d42f61640 10 monclient: monmap: 2023-11-25T05:27:09.370 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-11-25T05:27:09.370 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-25T05:27:09.370 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-11-25T05:27:09.370 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-11-25T05:27:09.370 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-11-25T05:27:09.371 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:27:09.371 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-11-25T05:27:09.371 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.noname-b 2023-11-25T05:27:09.371 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-11-25T05:27:09.371 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:27:09.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.364+0000 7f6d42f61640 1 librados: starting msgr at 2023-11-25T05:27:09.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.364+0000 7f6d42f61640 1 librados: starting objecter 2023-11-25T05:27:09.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.364+0000 7f6d42f61640 1 Processor -- start 2023-11-25T05:27:09.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.364+0000 7f6d42f61640 1 -- start start 2023-11-25T05:27:09.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.364+0000 7f6d42f61640 1 librados: setting wanted keys 2023-11-25T05:27:09.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.364+0000 7f6d42f61640 1 librados: calling monclient init 2023-11-25T05:27:09.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.364+0000 7f6d42f61640 10 monclient: init 2023-11-25T05:27:09.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 10 monclient: _reopen_session rank -1 2023-11-25T05:27:09.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-25T05:27:09.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c0a8930 0x7f6d3c148390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:27:09.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 10 monclient(hunting): picked mon.noname-c con 0x7f6d3c0a8930 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-11-25T05:27:09.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d3c13f170 0x7f6d3c148f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:27:09.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 10 monclient(hunting): picked mon.noname-a con 0x7f6d3c13f170 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-11-25T05:27:09.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3c145920 0x7f6d3c1d7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:27:09.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 10 monclient(hunting): picked mon.noname-b con 0x7f6d3c145920 addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2023-11-25T05:27:09.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 10 monclient(hunting): start opening mon connection 2023-11-25T05:27:09.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c14a580 con 0x7f6d3c13f170 2023-11-25T05:27:09.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 10 monclient(hunting): start opening mon connection 2023-11-25T05:27:09.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c147690 con 0x7f6d3c145920 2023-11-25T05:27:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 10 monclient(hunting): start opening mon connection 2023-11-25T05:27:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c1d73e0 con 0x7f6d3c0a8930 2023-11-25T05:27:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 10 monclient(hunting): _renew_subs 2023-11-25T05:27:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d42f61640 10 monclient(hunting): authenticate will time out at 2755.608179s 2023-11-25T05:27:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d3bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d3c13f170 0x7f6d3c148f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:27:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.365+0000 7f6d40cfd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c0a8930 0x7f6d3c148390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:27:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d414fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3c145920 0x7f6d3c1d7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:27:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d40cfd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c0a8930 0x7f6d3c148390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:42318/0 (socket says 172.21.15.7:42318) 2023-11-25T05:27:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d40cfd640 1 -- 172.21.15.7:0/3655884510 learned_addr learned my addr 172.21.15.7:0/3655884510 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-11-25T05:27:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d40cfd640 10 monclient(hunting): get_auth_request con 0x7f6d3c0a8930 auth_method 0 2023-11-25T05:27:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d40cfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:27:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d40cfd640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:27:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d40cfd640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:27:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d3bfff640 10 monclient(hunting): get_auth_request con 0x7f6d3c13f170 auth_method 0 2023-11-25T05:27:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:27:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d3bfff640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:27:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d3bfff640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:27:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d414fe640 10 monclient(hunting): get_auth_request con 0x7f6d3c145920 auth_method 0 2023-11-25T05:27:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d414fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-25T05:27:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d414fe640 10 monclient(hunting): _init_auth method 2 2023-11-25T05:27:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d414fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-25T05:27:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d40cfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:27:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d40cfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:27:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d40cfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:27:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:27:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:27:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:27:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d414fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-25T05:27:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d414fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-25T05:27:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.366+0000 7f6d414fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-25T05:27:09.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.369+0000 7f6d414fe640 10 monclient(hunting): handle_auth_done global_id 21325 payload 824 2023-11-25T05:27:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.369+0000 7f6d414fe640 1 -- 172.21.15.7:0/3655884510 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c0a8930 msgr2=0x7f6d3c148390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:27:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.369+0000 7f6d414fe640 1 --2- 172.21.15.7:0/3655884510 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c0a8930 0x7f6d3c148390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.369+0000 7f6d414fe640 1 -- 172.21.15.7:0/3655884510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d3c13f170 msgr2=0x7f6d3c148f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:27:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.369+0000 7f6d414fe640 1 --2- 172.21.15.7:0/3655884510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d3c13f170 0x7f6d3c148f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.369+0000 7f6d414fe640 10 monclient: _finish_hunting 0 2023-11-25T05:27:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.369+0000 7f6d414fe640 1 monclient: found mon.noname-b 2023-11-25T05:27:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.369+0000 7f6d414fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-25T05:27:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.369+0000 7f6d414fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.90:3300/0 2023-11-25T05:27:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.369+0000 7f6d414fe640 1 -- 172.21.15.7:0/3655884510 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d34009a70 con 0x7f6d3c145920 2023-11-25T05:27:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.369+0000 7f6d40cfd640 1 --2- 172.21.15.7:0/3655884510 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c0a8930 0x7f6d3c148390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-25T05:27:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.371+0000 7f6d414fe640 1 --2- 172.21.15.7:0/3655884510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3c145920 0x7f6d3c1d7900 secure :-1 s=READY pgs=5179 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c00c340 tx=0x7f6d2c001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:27:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.371+0000 7f6d397fa640 10 client.?.objecter ms_handle_connect 0x7f6d3c145920 2023-11-25T05:27:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.371+0000 7f6d397fa640 10 client.?.objecter resend_mon_ops 2023-11-25T05:27:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d397fa640 1 -- 172.21.15.7:0/3655884510 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c005ce0 con 0x7f6d3c145920 2023-11-25T05:27:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:27:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d397fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-25T05:27:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d397fa640 10 monclient: dump: 2023-11-25T05:27:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:27:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:27:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:27:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:27:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:27:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:27:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:27:09.379 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:27:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d397fa640 10 monclient: _finish_auth 0 2023-11-25T05:27:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d397fa640 10 monclient: _check_auth_tickets 2023-11-25T05:27:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-25T05:27:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d397fa640 1 -- 172.21.15.7:0/3655884510 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2c005e40 con 0x7f6d3c145920 2023-11-25T05:27:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d397fa640 10 monclient: handle_config config(0 keys) v1 2023-11-25T05:27:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d397fa640 1 -- 172.21.15.7:0/3655884510 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c0072d0 con 0x7f6d3c145920 2023-11-25T05:27:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-25T05:27:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-25T05:27:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d397fa640 10 monclient: dump: 2023-11-25T05:27:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-11-25T05:27:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 59cf1eff-816d-4eda-b513-a2de7ea218f7 2023-11-25T05:27:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-11-25T05:02:46.536366+0000 2023-11-25T05:27:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2023-11-25T05:27:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-11-25T05:27:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-25T05:27:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.b 2023-11-25T05:27:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-11-25T05:27:09.380 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:27:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d42f61640 5 monclient: authenticate success, global_id 21325 2023-11-25T05:27:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d42f61640 10 monclient: _renew_subs 2023-11-25T05:27:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d42f61640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:27:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d42f61640 1 -- 172.21.15.7:0/3655884510 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d24009400 con 0x7f6d3c145920 2023-11-25T05:27:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d3affd640 4 set_mon_vals no callback set 2023-11-25T05:27:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d42f61640 10 client.21325.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-25T05:27:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d42f61640 10 monclient: _renew_subs 2023-11-25T05:27:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d42f61640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:27:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d42f61640 1 -- 172.21.15.7:0/3655884510 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d3c1d7e40 con 0x7f6d3c145920 2023-11-25T05:27:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.375+0000 7f6d42f61640 1 librados: init done 2023-11-25T05:27:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.376+0000 7f6d42f61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-25T05:27:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.376+0000 7f6d42f61640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:27:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.376+0000 7f6d42f61640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:27:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.376+0000 7f6d42f61640 1 -- 172.21.15.7:0/3655884510 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d3c1d7250 con 0x7f6d3c145920 2023-11-25T05:27:09.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.394+0000 7f6d397fa640 1 -- 172.21.15.7:0/3655884510 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6d2c007430 con 0x7f6d3c145920 2023-11-25T05:27:09.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.394+0000 7f6d397fa640 1 --2- 172.21.15.7:0/3655884510 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6d1c06c820 0x7f6d1c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-25T05:27:09.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.394+0000 7f6d40cfd640 1 --2- 172.21.15.7:0/3655884510 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6d1c06c820 0x7f6d1c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-25T05:27:09.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.394+0000 7f6d40cfd640 10 monclient: get_auth_request con 0x7f6d1c06c820 auth_method 0 2023-11-25T05:27:09.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.395+0000 7f6d397fa640 10 client.21325.objecter ms_handle_connect 0x7f6d1c06c820 2023-11-25T05:27:09.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.395+0000 7f6d40cfd640 1 --2- 172.21.15.7:0/3655884510 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6d1c06c820 0x7f6d1c06ece0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6d34003e80 tx=0x7f6d3400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-25T05:27:09.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.396+0000 7f6d397fa640 1 -- 172.21.15.7:0/3655884510 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(918..918 src has 687..918) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f6d2c057db0 con 0x7f6d3c145920 2023-11-25T05:27:09.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.396+0000 7f6d397fa640 10 client.21325.objecter ms_dispatch 0x7f6d3c000b90 osd_map(918..918 src has 687..918) v4 2023-11-25T05:27:09.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.396+0000 7f6d397fa640 3 client.21325.objecter handle_osd_map got epochs [918,918] > 0 2023-11-25T05:27:09.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.396+0000 7f6d397fa640 3 client.21325.objecter handle_osd_map decoding full epoch 918 2023-11-25T05:27:09.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.396+0000 7f6d397fa640 20 client.21325.objecter dump_active .. 0 homeless 2023-11-25T05:27:09.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.627+0000 7f6d397fa640 1 -- 172.21.15.7:0/3655884510 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d2c0540f0 con 0x7f6d3c145920 2023-11-25T05:27:09.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.627+0000 7f6d397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-25T05:27:09.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.627+0000 7f6d397fa640 10 monclient: _finish_command 1 = system:0 2023-11-25T05:27:09.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.805+0000 7f6d42f61640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-11-25T05:27:09.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.805+0000 7f6d42f61640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-11-25T05:27:09.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.805+0000 7f6d42f61640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.90:3300/0 2023-11-25T05:27:09.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:09.805+0000 7f6d42f61640 1 -- 172.21.15.7:0/3655884510 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f6d3c0aa400 con 0x7f6d3c145920 2023-11-25T05:27:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.037+0000 7f6d397fa640 1 -- 172.21.15.7:0/3655884510 <== mon.1 v2:172.21.15.90:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f6d2c05be50 con 0x7f6d3c145920 2023-11-25T05:27:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.037+0000 7f6d397fa640 10 monclient: handle_config config(1 keys) v1 2023-11-25T05:27:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.038+0000 7f6d3affd640 4 set_mon_vals no callback set 2023-11-25T05:27:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.038+0000 7f6d397fa640 1 -- 172.21.15.7:0/3655884510 <== mon.1 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7f6d2c057710 con 0x7f6d3c145920 2023-11-25T05:27:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.038+0000 7f6d397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-11-25T05:27:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.038+0000 7f6d3affd640 10 set_mon_vals mon_health_to_clog = false 2023-11-25T05:27:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.038+0000 7f6d397fa640 10 monclient: _finish_command 2 = system:0 2023-11-25T05:27:10.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 10 librados: watch_flush enter 2023-11-25T05:27:10.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 10 librados: watch_flush exit 2023-11-25T05:27:10.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 20 client.21325.objecter shutdown clearing up homeless session... 2023-11-25T05:27:10.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 10 client.21325.objecter successfully canceled tick 2023-11-25T05:27:10.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 1 -- 172.21.15.7:0/3655884510 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6d1c06c820 msgr2=0x7f6d1c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:27:10.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 1 --2- 172.21.15.7:0/3655884510 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6d1c06c820 0x7f6d1c06ece0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6d34003e80 tx=0x7f6d3400b040 comp rx=0 tx=0).stop 2023-11-25T05:27:10.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 10 monclient: shutdown 2023-11-25T05:27:10.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-25T05:27:10.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 1 -- 172.21.15.7:0/3655884510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3c145920 msgr2=0x7f6d3c1d7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-25T05:27:10.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 1 --2- 172.21.15.7:0/3655884510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3c145920 0x7f6d3c1d7900 secure :-1 s=READY pgs=5179 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c00c340 tx=0x7f6d2c001e70 comp rx=0 tx=0).stop 2023-11-25T05:27:10.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 1 -- 172.21.15.7:0/3655884510 shutdown_connections 2023-11-25T05:27:10.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 1 --2- 172.21.15.7:0/3655884510 >> [v2:172.21.15.90:6800/1055092586,v1:172.21.15.90:6801/1055092586] conn(0x7f6d1c06c820 0x7f6d1c06ece0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:10.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 1 --2- 172.21.15.7:0/3655884510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3c145920 0x7f6d3c1d7900 unknown :-1 s=CLOSED pgs=5179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:10.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 1 --2- 172.21.15.7:0/3655884510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d3c13f170 0x7f6d3c148f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:10.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 1 --2- 172.21.15.7:0/3655884510 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c0a8930 0x7f6d3c148390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-25T05:27:10.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.040+0000 7f6d42f61640 1 -- 172.21.15.7:0/3655884510 >> 172.21.15.7:0/3655884510 conn(0x7f6d3c0619c0 msgr2=0x7f6d3c13d810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-25T05:27:10.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.041+0000 7f6d42f61640 1 -- 172.21.15.7:0/3655884510 shutdown_connections 2023-11-25T05:27:10.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.041+0000 7f6d42f61640 1 -- 172.21.15.7:0/3655884510 wait complete. 2023-11-25T05:27:10.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-25T05:27:10.041+0000 7f6d42f61640 1 librados: shutdown 2023-11-25T05:27:10.054 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-25T05:27:10.054 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-25T05:27:10.055 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-25T05:27:10.055 INFO:teuthology.orchestra.run:waiting for 300 2023-11-25T05:27:10.072 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-25T05:27:10.058+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 87133) UID: 0 2023-11-25T05:27:10.072 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-11-25T05:27:10.060+0000 b47e640 -1 osd.0 918 *** Got signal Terminated *** 2023-11-25T05:27:16.751 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:58.949 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:16.752 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:58.951 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:16.768 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:58.966 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:16.798 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:58.942 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:16.799 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:58.944 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:16.800 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:58.998 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:16.801 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:59.000 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:16.814 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:23:58.958 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:16.817 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:59.015 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:26.640 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:08.838 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:26.644 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:08.842 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:26.655 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:08.853 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:26.696 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:08.839 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:26.698 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:08.843 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:26.707 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:08.905 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:26.709 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:08.853 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:26.711 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:08.909 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:26.725 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:08.923 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:36.667 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:18.865 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:36.668 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:18.867 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:36.687 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:18.885 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:36.716 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:18.860 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:36.721 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:18.863 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:36.721 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:18.919 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:36.724 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:18.922 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:36.731 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:18.875 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:36.741 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:18.939 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:46.283 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:28.481 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:46.291 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:28.489 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:46.318 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:28.516 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:46.353 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:28.497 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:46.354 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:28.552 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:46.354 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:28.499 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:46.355 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:28.554 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:46.371 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:28.515 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:46.375 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:28.573 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:46.910 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:29.054 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:46.911 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:29.055 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:46.967 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:29.111 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:47.044 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:29.243 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:47.046 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:29.244 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:47.052 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:29.251 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:47.053 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:29.252 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:47.054 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:29.252 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:47.062 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:29.260 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:56.768 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:38.966 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:56.770 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:38.969 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:56.784 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:38.982 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:56.816 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:38.961 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:56.817 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:39.015 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:56.818 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:38.962 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:56.821 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:39.019 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:56.831 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:38.975 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:27:56.834 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:39.032 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:06.795 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:48.993 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:06.799 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:48.998 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:06.815 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:49.013 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:06.847 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:49.045 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:06.848 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:48.992 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:06.850 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:48.994 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:06.852 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:49.050 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:06.861 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:49.005 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:06.865 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:49.063 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:16.819 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:59.017 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:16.821 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:59.019 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:16.837 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:59.035 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:16.871 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:59.015 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:16.872 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:59.016 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:16.883 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:24:59.027 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:16.884 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:59.082 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:16.885 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:59.083 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:16.897 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:59.095 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:26.839 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:09.037 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:26.844 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:09.042 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:26.861 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:09.059 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:26.891 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:09.035 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:26.892 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:09.036 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:26.898 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:09.096 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:26.899 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:09.097 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:26.904 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:09.048 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:26.910 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:09.108 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:36.819 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:19.017 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:36.867 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:19.066 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:36.882 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:19.080 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:36.917 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:19.062 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:36.919 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:19.117 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:36.920 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:19.065 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:36.923 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:19.121 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:36.934 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:19.078 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:36.937 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:19.135 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:46.891 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:29.089 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:46.895 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:29.093 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:46.909 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:29.108 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:46.942 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:29.086 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:46.944 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:29.089 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:46.956 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:29.154 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:46.956 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:29.100 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:46.957 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:29.156 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:46.969 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:29.167 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:56.928 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:39.126 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:56.931 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:39.129 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:56.948 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:39.146 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:56.981 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:39.125 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:56.983 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:39.127 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:56.998 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:39.142 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:57.000 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:39.198 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:57.002 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:39.200 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:57.025 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:39.223 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:28:59.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-11-25T05:28:59.881 INFO:tasks.ceph.osd.0:Stopped 2023-11-25T05:28:59.881 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-25T05:28:59.881 INFO:teuthology.orchestra.run:waiting for 300 2023-11-25T05:28:59.900 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-25T05:28:59.883+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 87135) UID: 0 2023-11-25T05:28:59.900 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-11-25T05:28:59.885+0000 b47e640 -1 osd.1 926 *** Got signal Terminated *** 2023-11-25T05:29:02.058 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:44.256 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:02.061 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:44.259 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:02.075 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:44.274 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:02.113 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:44.258 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:02.115 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:44.259 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:02.126 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:44.270 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:02.135 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:44.333 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:02.136 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:44.335 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:02.148 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:44.346 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:05.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-11-25T05:29:16.999 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:59.197 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:16.999 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:59.198 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:17.013 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:59.211 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:17.047 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:59.191 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:17.048 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:59.246 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:17.048 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:59.193 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:17.049 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:59.248 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:17.063 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:25:59.208 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:17.064 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:59.262 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:22.011 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:04.209 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:22.015 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:04.213 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:22.030 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:04.229 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:22.062 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:04.260 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:22.063 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:04.207 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:22.065 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:04.263 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:22.066 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:04.210 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:22.076 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:04.274 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:22.076 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:04.221 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:37.048 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:19.246 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:37.052 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:19.250 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:37.069 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:19.267 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:37.103 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:19.247 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:37.105 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:19.249 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:37.107 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:19.306 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:37.110 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:19.308 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:37.125 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:19.269 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:37.127 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:19.325 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:42.062 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:24.260 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:42.065 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:24.263 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:42.082 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:24.280 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:42.113 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:24.257 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:42.115 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:24.259 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:42.116 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:24.314 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:42.119 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:24.317 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:42.124 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:24.269 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:42.133 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:24.332 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:57.102 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:39.300 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:57.106 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:39.305 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:57.125 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:39.323 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:57.158 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:39.302 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:57.160 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:39.358 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:57.161 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:39.306 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:57.163 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:39.362 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:57.176 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:39.320 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:29:57.177 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:39.376 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:02.117 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:44.315 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:02.120 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:44.318 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:02.133 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:44.331 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:02.166 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:44.310 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:02.169 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:44.313 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:02.172 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:44.371 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:02.174 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:44.373 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:02.185 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:44.328 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:02.185 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:44.382 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:17.159 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:59.357 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:17.163 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:59.361 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:17.180 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:59.378 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:17.211 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:59.356 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:17.212 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:59.357 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:17.213 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:59.411 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:17.216 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:59.415 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:17.229 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:26:59.372 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:17.235 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:59.434 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:22.172 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:04.370 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:22.175 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:04.373 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:22.190 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:04.388 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:22.223 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:04.368 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:22.225 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:04.423 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:22.227 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:04.371 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:22.228 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:04.426 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:22.237 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:04.381 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:22.239 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:04.437 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:37.214 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:19.413 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:37.217 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:19.415 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:37.236 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:19.434 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:37.270 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:19.414 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:37.272 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:19.416 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:37.289 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:19.432 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:37.295 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:19.494 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:37.298 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:19.496 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:37.312 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:19.511 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:42.182 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:24.380 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:42.229 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:24.373 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:42.232 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:24.430 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:42.232 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:24.377 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:42.245 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:24.389 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:42.248 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:24.446 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:42.282 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:24.480 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:42.285 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:24.483 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:42.296 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:24.494 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:54.051 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:36.249 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:54.102 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:36.246 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:54.103 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:36.248 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:54.104 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:36.302 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:54.107 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:36.306 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:54.112 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:36.310 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:54.120 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:36.264 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:54.124 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:36.322 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:54.129 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:36.328 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:57.267 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:39.465 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:57.269 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:39.468 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:57.282 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:39.480 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:57.317 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:39.461 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:57.320 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:39.518 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:57.321 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:39.466 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:57.323 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:39.521 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:57.332 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:39.476 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:30:57.334 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:39.533 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:02.281 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:44.479 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:02.288 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:44.486 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:02.300 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:44.499 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:02.335 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:44.533 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:02.336 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:44.480 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:02.338 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:44.536 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:02.338 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:44.483 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:02.349 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:44.547 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:02.350 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:44.494 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:17.328 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:59.526 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:17.330 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:59.529 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:17.344 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:59.542 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:17.378 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:59.522 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:17.379 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:59.524 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:17.394 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:27:59.538 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:17.399 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:59.597 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:17.403 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:59.602 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:17.420 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:59.618 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:22.337 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:04.535 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:22.339 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:04.537 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:22.351 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:04.549 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:22.390 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:04.588 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:22.391 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:04.535 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:22.392 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:04.590 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:22.393 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:04.538 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:22.404 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:04.548 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:22.407 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:04.605 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:37.385 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:19.584 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:37.387 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:19.585 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:37.400 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:19.598 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:37.436 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:19.580 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:37.437 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:19.581 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:37.451 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:19.595 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:37.452 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:19.650 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:37.453 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:19.652 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:37.474 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:19.671 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:42.412 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:24.610 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:42.414 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:24.613 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:42.424 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:24.622 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:42.460 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:24.604 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:42.461 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:24.606 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:42.464 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:24.662 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:42.465 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:24.664 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:42.472 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:24.616 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:42.479 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:24.677 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:57.491 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:39.635 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:57.491 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:39.689 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:57.494 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:39.638 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:57.494 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:39.693 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:57.509 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:39.654 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:57.510 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:39.708 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:57.545 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:39.743 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:57.546 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:39.745 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:31:57.561 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:39.759 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:02.482 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:44.680 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:02.487 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:44.686 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:02.499 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:44.698 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:02.534 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:44.679 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:02.535 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:44.733 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:02.538 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:44.682 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:02.538 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:44.737 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:02.549 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:44.693 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:02.549 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:44.748 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:17.532 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:59.730 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:17.536 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:59.735 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:17.558 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:59.756 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:17.585 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:59.729 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:17.586 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:59.731 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:17.591 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:59.790 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:17.597 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:59.795 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:17.600 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:28:59.745 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:17.612 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:59.810 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:22.538 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:04.734 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:22.538 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:04.736 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:22.555 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:04.752 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:22.589 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:04.733 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:22.590 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:04.734 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:22.591 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:04.789 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:22.593 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:04.791 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:22.600 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:04.744 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:22.605 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:04.804 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:37.617 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:19.815 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:37.619 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:19.817 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:37.637 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:19.835 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:37.666 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:19.810 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:37.667 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:19.811 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:37.667 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:19.866 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:37.681 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:19.826 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:37.683 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:19.882 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:37.699 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:19.897 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:42.605 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:24.803 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:42.608 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:24.806 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:42.621 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:24.820 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:42.658 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:24.856 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:42.659 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:24.857 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:42.660 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:24.804 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:42.661 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:24.806 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:42.669 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:24.867 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:42.671 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:24.816 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:57.654 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:39.851 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:57.655 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:39.852 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:57.672 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:39.870 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:57.712 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:39.911 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:57.714 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:39.912 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:57.718 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:39.862 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:57.719 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:39.863 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:57.729 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:39.927 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:32:57.732 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:39.877 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:02.665 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:44.863 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:02.667 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:44.865 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:02.677 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:44.875 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:02.714 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:44.858 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:02.717 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:44.861 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:02.730 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:44.874 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:02.739 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:44.938 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:02.743 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:44.942 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:02.755 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:44.954 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:17.704 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:59.902 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:17.706 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:59.904 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:17.719 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:59.917 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:17.753 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:59.897 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:17.755 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:59.954 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:17.756 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:59.900 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:17.757 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:59.955 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:17.771 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:29:59.915 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:17.776 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:59.975 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:22.717 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:04.915 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:22.720 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:04.918 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:22.731 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:04.928 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:22.766 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:04.964 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:22.769 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:04.967 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:22.771 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:04.916 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:22.772 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:04.917 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:22.780 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:04.978 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:22.783 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:04.927 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:37.753 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:19.951 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:37.757 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:19.955 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:37.778 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:19.976 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:37.807 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:20.005 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:37.808 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:19.952 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:37.810 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:20.008 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:37.812 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:19.957 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:37.825 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:20.023 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:37.827 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:19.971 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:42.768 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:24.966 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:42.771 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:24.969 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:42.787 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:24.985 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:42.820 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:24.964 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:42.822 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:24.966 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:42.823 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:25.021 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:42.828 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:25.026 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:42.833 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:24.977 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:42.840 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:25.038 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:57.839 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:40.037 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:57.842 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:40.040 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:57.882 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:40.080 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:57.892 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:40.036 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:57.892 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:40.091 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:57.893 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:40.038 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:57.897 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:40.095 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:57.910 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:40.055 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:57.923 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:40.121 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:58.366 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:40.511 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:58.397 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:40.595 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:58.398 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:40.596 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:58.409 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:40.608 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:58.418 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:40.562 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:58.426 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:40.569 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:58.438 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:40.636 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:58.439 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:40.638 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:58.448 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:40.646 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:33:58.850 INFO:tasks.ceph.osd.1:Stopped 2023-11-25T05:33:58.851 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-25T05:33:58.851 INFO:teuthology.orchestra.run:waiting for 300 2023-11-25T05:33:58.864 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:33:58.854+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 87137) UID: 0 2023-11-25T05:33:58.864 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-11-25T05:33:58.856+0000 b47e640 -1 osd.2 928 *** Got signal Terminated *** 2023-11-25T05:34:02.299 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:44.498 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:02.304 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:44.502 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:02.321 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:44.519 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:02.351 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:44.495 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:02.355 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:44.499 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:02.361 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:44.560 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:02.363 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:44.562 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:02.375 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:44.519 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:02.381 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:44.579 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:03.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-11-25T05:34:04.953 INFO:tasks.ceph.osd.2:Stopped 2023-11-25T05:34:04.954 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-11-25T05:34:04.954 INFO:teuthology.orchestra.run:waiting for 300 2023-11-25T05:34:04.968 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-11-25T05:34:04.957+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 87139) UID: 0 2023-11-25T05:34:04.968 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-11-25T05:34:04.959+0000 b47e640 -1 osd.3 932 *** Got signal Terminated *** 2023-11-25T05:34:09.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-11-25T05:34:09.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-11-25T05:34:11.055 INFO:tasks.ceph.osd.3:Stopped 2023-11-25T05:34:11.056 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-11-25T05:34:11.056 INFO:teuthology.orchestra.run:waiting for 300 2023-11-25T05:34:11.070 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:34:11.059+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86828) UID: 0 2023-11-25T05:34:11.072 INFO:tasks.ceph.osd.4.smithi090.stderr:2023-11-25T05:34:11.061+0000 b47e640 -1 osd.4 934 *** Got signal Terminated *** 2023-11-25T05:34:13.047 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:55.245 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:13.048 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:55.246 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:13.065 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:55.263 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:13.101 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:55.300 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:13.103 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:55.301 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:13.103 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:55.247 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:13.106 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:55.250 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:13.123 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:30:55.267 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:13.125 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:55.323 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:15.334 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-11-25T05:34:17.936 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:00.134 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:17.938 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:00.136 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:17.948 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:00.147 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:17.980 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:00.124 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:17.983 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:00.127 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:17.993 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:00.137 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:17.999 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:00.197 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:18.001 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:00.200 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:18.015 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:00.213 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:32.980 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:15.179 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:32.985 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:15.183 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:33.001 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:15.199 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:33.033 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:15.178 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:33.035 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:15.179 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:33.035 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:15.233 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:33.037 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:15.235 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:33.049 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:15.193 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:33.053 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:15.251 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:37.998 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:20.196 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:38.002 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:20.200 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:38.017 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:20.215 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:38.048 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:20.193 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:38.049 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:20.194 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:38.054 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:20.252 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:38.059 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:20.257 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:38.059 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:20.204 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:38.070 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:20.268 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:53.017 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:35.215 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:53.072 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:35.270 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:53.089 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:35.288 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:53.120 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:35.265 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:53.121 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:35.266 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:53.135 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:35.279 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:53.147 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:35.345 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:53.149 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:35.348 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:53.165 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:35.363 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:58.117 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:40.315 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:58.121 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:40.320 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:58.134 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:40.333 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:58.168 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:40.312 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:58.170 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:40.368 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:58.171 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:40.315 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:58.172 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:40.369 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:58.182 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:40.325 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:34:58.183 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:40.381 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:13.142 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:55.340 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:13.144 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:55.342 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:13.158 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:55.357 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:13.193 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:55.392 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:13.194 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:55.393 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:13.196 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:55.340 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:13.196 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:55.341 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:13.210 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:31:55.355 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:13.211 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:55.410 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:18.150 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:00.348 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:18.153 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:00.351 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:18.163 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:00.361 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:18.196 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:32:00.341 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:18.198 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:32:00.343 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:18.201 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:00.399 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:18.202 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:00.400 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:18.209 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:32:00.353 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:18.215 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:00.413 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:26.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-11-25T05:35:30.376 INFO:tasks.ceph.osd.4:Stopped 2023-11-25T05:35:30.377 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-11-25T05:35:30.377 INFO:teuthology.orchestra.run:waiting for 300 2023-11-25T05:35:30.390 INFO:tasks.ceph.osd.5.smithi090.stderr:2023-11-25T05:35:30.380+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86831) UID: 0 2023-11-25T05:35:30.392 INFO:tasks.ceph.osd.5.smithi090.stderr:2023-11-25T05:35:30.381+0000 b47e640 -1 osd.5 936 *** Got signal Terminated *** 2023-11-25T05:35:31.932 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-11-25T05:35:33.197 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:15.395 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:33.199 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:15.397 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:33.226 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:15.424 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:33.243 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:32:15.387 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:33.245 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:32:15.390 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:33.259 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:15.457 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:33.260 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:15.459 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:33.262 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:32:15.407 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:33.277 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:15.476 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:36.479 INFO:tasks.ceph.osd.5:Stopped 2023-11-25T05:35:36.479 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-11-25T05:35:36.479 INFO:teuthology.orchestra.run:waiting for 300 2023-11-25T05:35:36.492 INFO:tasks.ceph.osd.6.smithi090.stderr:2023-11-25T05:35:36.482+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86833) UID: 0 2023-11-25T05:35:36.492 INFO:tasks.ceph.osd.6.smithi090.stderr:2023-11-25T05:35:36.483+0000 b47e640 -1 osd.6 938 *** Got signal Terminated *** 2023-11-25T05:35:38.209 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:20.407 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:38.213 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:20.411 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:38.223 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:20.421 86850== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:38.257 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:32:20.402 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:38.259 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:32:20.403 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:38.261 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:20.460 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:38.263 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:20.461 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:38.268 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:32:20.413 86215== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:38.274 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:20.473 86853== Warning: unimplemented fcntl command: 1036 2023-11-25T05:35:42.581 INFO:tasks.ceph.osd.6:Stopped 2023-11-25T05:35:42.582 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-11-25T05:35:42.582 INFO:teuthology.orchestra.run:waiting for 300 2023-11-25T05:35:42.594 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-25T05:35:42.585+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86835) UID: 0 2023-11-25T05:35:42.595 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-25T05:35:42.586+0000 b47e640 -1 osd.7 938 *** Got signal Terminated *** 2023-11-25T05:35:48.683 INFO:tasks.ceph.osd.7:Stopped 2023-11-25T05:35:48.684 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-25T05:35:48.684 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-11-25T05:35:48.684 INFO:teuthology.orchestra.run:waiting for 300 2023-11-25T05:35:48.708 INFO:tasks.ceph.mgr.y:Stopped 2023-11-25T05:35:48.708 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-25T05:35:48.708 INFO:teuthology.orchestra.run:waiting for 300 2023-11-25T05:35:48.709 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-25T05:35:48.708+0000 7fdcc8b09640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 86213) UID: 0 2023-11-25T05:35:48.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-11-25T05:35:48.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-11-25T05:35:48.738 INFO:tasks.ceph.mgr.x:Stopped 2023-11-25T05:35:48.739 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-25T05:35:48.739 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-25T05:35:48.739 INFO:teuthology.orchestra.run:waiting for 300 2023-11-25T05:35:48.762 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-11-25T05:35:48.751+0000 9f58640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 86816) UID: 0 2023-11-25T05:35:48.763 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-11-25T05:35:48.753+0000 9f58640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-11-25T05:35:49.600 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:31.798 86850== 2023-11-25T05:35:49.600 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:31.798 86850== Exit program on first error (--exit-on-first-error=yes) 2023-11-25T05:35:49.622 INFO:tasks.ceph.mon.a.smithi007.stderr:daemon-helper: command failed with exit status 42 2023-11-25T05:35:53.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-11-25T05:35:53.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-11-25T05:35:53.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2023-11-25T05:35:54.841 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-11-25T05:35:54.841 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-11-25T05:35:54.842 INFO:tasks.ceph.mon.a:Stopped 2023-11-25T05:35:54.843 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-25T05:35:54.843 INFO:teuthology.orchestra.run:waiting for 300 2023-11-25T05:35:54.858 INFO:tasks.ceph.mon.c.smithi007.stderr:2023-11-25T05:35:54.846+0000 9f58640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 86818) UID: 0 2023-11-25T05:35:54.859 INFO:tasks.ceph.mon.c.smithi007.stderr:2023-11-25T05:35:54.848+0000 9f58640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-11-25T05:35:55.471 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:37.669 86853== 2023-11-25T05:35:55.471 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:37.669 86853== Exit program on first error (--exit-on-first-error=yes) 2023-11-25T05:35:55.488 INFO:tasks.ceph.mon.c.smithi007.stderr:daemon-helper: command failed with exit status 42 2023-11-25T05:35:59.043 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-11-25T05:35:59.043 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-11-25T05:36:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-11-25T05:36:00.945 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-11-25T05:36:00.945 INFO:tasks.ceph.mon.c:Stopped 2023-11-25T05:36:00.946 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-25T05:36:00.946 INFO:teuthology.orchestra.run:waiting for 300 2023-11-25T05:36:00.961 INFO:tasks.ceph.mon.b.smithi090.stderr:2023-11-25T05:36:00.949+0000 9f58640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 86211) UID: 0 2023-11-25T05:36:00.961 INFO:tasks.ceph.mon.b.smithi090.stderr:2023-11-25T05:36:00.951+0000 9f58640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2023-11-25T05:36:01.574 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:32:43.718 86215== 2023-11-25T05:36:01.574 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:32:43.718 86215== Exit program on first error (--exit-on-first-error=yes) 2023-11-25T05:36:01.590 INFO:tasks.ceph.mon.b.smithi090.stderr:daemon-helper: command failed with exit status 42 2023-11-25T05:36:04.044 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-11-25T05:36:04.045 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-11-25T05:36:07.048 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-11-25T05:36:07.048 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-11-25T05:36:07.049 INFO:tasks.ceph.mon.b:Stopped 2023-11-25T05:36:07.049 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-25T05:36:07.049 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-11-25T05:36:07.093 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-25T05:36:07.093 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-25T05:36:07.201 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-25T05:36:07.202 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-25T05:36:07.283 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-25T05:36:07.284 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-25T05:36:07.350 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-25T05:36:07.350 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-11-25T05:36:07.425 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-25T05:36:07.425 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-11-25T05:36:07.599 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-25T05:36:07.599 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-11-25T05:36:07.720 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-25T05:36:07.721 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-11-25T05:36:07.776 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-25T05:36:07.776 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-11-25T05:36:07.835 INFO:tasks.ceph:Archiving mon data... 2023-11-25T05:36:07.837 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196/data/mon.a.tgz 2023-11-25T05:36:07.837 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2023-11-25T05:36:07.857 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.VtiMvYK2VK 2023-11-25T05:36:07.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.VtiMvYK2VK 2023-11-25T05:36:08.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.VtiMvYK2VK 2023-11-25T05:36:08.117 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.VtiMvYK2VK is 425KB 2023-11-25T05:36:08.216 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.VtiMvYK2VK 2023-11-25T05:36:08.232 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196/data/mon.c.tgz 2023-11-25T05:36:08.232 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2023-11-25T05:36:08.287 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.geCmMaPazC 2023-11-25T05:36:08.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.geCmMaPazC 2023-11-25T05:36:08.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.geCmMaPazC 2023-11-25T05:36:08.524 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.geCmMaPazC is 426KB 2023-11-25T05:36:08.620 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.geCmMaPazC 2023-11-25T05:36:08.634 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196/data/mon.b.tgz 2023-11-25T05:36:08.635 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2023-11-25T05:36:08.652 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.XQnUGilzLH 2023-11-25T05:36:08.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.XQnUGilzLH 2023-11-25T05:36:08.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.XQnUGilzLH 2023-11-25T05:36:08.898 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.XQnUGilzLH is 426KB 2023-11-25T05:36:09.002 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.XQnUGilzLH 2023-11-25T05:36:09.016 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-25T05:36:09.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-25T05:36:09.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-25T05:36:09.045 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-11-25T05:36:09.082 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-25T05:36:09.083 DEBUG:teuthology.orchestra.run.smithi007:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-25T05:36:09.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-25T05:36:09.136 INFO:teuthology.orchestra.run.smithi007.stdout:/var/log/ceph/valgrind/mon.a.log: Leak_StillReachable 2023-11-25T05:36:09.136 INFO:teuthology.orchestra.run.smithi007.stdout:/var/log/ceph/valgrind/mon.c.log: Leak_StillReachable 2023-11-25T05:36:09.136 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind Leak_StillReachable 2023-11-25T05:36:09.136 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.a.log 2023-11-25T05:36:09.136 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind Leak_StillReachable 2023-11-25T05:36:09.136 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.c.log 2023-11-25T05:36:09.162 INFO:teuthology.orchestra.run.smithi090.stdout:/var/log/ceph/valgrind/mon.b.log: Leak_StillReachable 2023-11-25T05:36:09.163 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind Leak_StillReachable 2023-11-25T05:36:09.163 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.b.log 2023-11-25T05:36:09.163 INFO:tasks.ceph:Archiving crash dumps... 2023-11-25T05:36:09.165 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196/remote/smithi007/crash 2023-11-25T05:36:09.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-25T05:36:09.196 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196/remote/smithi090/crash 2023-11-25T05:36:09.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-25T05:36:09.228 INFO:tasks.ceph:Compressing logs... 2023-11-25T05:36:09.229 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-25T05:36:09.241 DEBUG:teuthology.orchestra.run.smithi090:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-25T05:36:09.269 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159424.log 2023-11-25T05:36:09.269 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92231.log 2023-11-25T05:36:09.294 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.159424.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.81007.log 2023-11-25T05:36:09.294 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92231.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-25T05:36:09.294 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.tmp-client.admin.81007.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.159424.log.gz 2023-11-25T05:36:09.294 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142322.log 2023-11-25T05:36:09.294 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.81007.log.gz 2023-11-25T05:36:09.294 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91893.log 2023-11-25T05:36:09.295 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.92231.log.gz 2023-11-25T05:36:09.295 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.142322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159214.log 2023-11-25T05:36:09.295 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92021.log 2023-11-25T05:36:09.295 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.159214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91853.log 2023-11-25T05:36:09.295 INFO:teuthology.orchestra.run.smithi007.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91893.log.gz 2023-11-25T05:36:09.295 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93468.log 2023-11-25T05:36:09.295 INFO:teuthology.orchestra.run.smithi007.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.159214.log.gz 2023-11-25T05:36:09.295 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86858.log 2023-11-25T05:36:09.295 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.92021.log.gz 2023-11-25T05:36:09.295 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87187.log 2023-11-25T05:36:09.295 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91853.log.gz 2023-11-25T05:36:09.295 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.86858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93323.log 2023-11-25T05:36:09.295 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.87187.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.142322.log.gz 2023-11-25T05:36:09.295 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2023-11-25T05:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.93323.log: -- /var/log/ceph/valgrind/osd.2.log 2023-11-25T05:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.93468.log.gz 2023-11-25T05:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87187.log.gz 2023-11-25T05:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2023-11-25T05:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2023-11-25T05:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.86858.log.gz 2023-11-25T05:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2023-11-25T05:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2023-11-25T05:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2023-11-25T05:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2023-11-25T05:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2023-11-25T05:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/valgrind/osd.1.log: /var/log/ceph/valgrind/mon.a.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2023-11-25T05:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91948.log 2023-11-25T05:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2023-11-25T05:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90967.log 2023-11-25T05:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2023-11-25T05:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110496.log 2023-11-25T05:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91948.log: /var/log/ceph/ceph-client.admin.90967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129425.log 2023-11-25T05:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91948.log.gz 2023-11-25T05:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:gzip/var/log/ceph/ceph-client.admin.110496.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.159551.log 2023-11-25T05:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90967.log.gz 2023-11-25T05:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.129425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159256.log 2023-11-25T05:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.159551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91509.log 2023-11-25T05:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.110496.log.gz 2023-11-25T05:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.93323.log.gz 2023-11-25T05:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.159256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92758.log 2023-11-25T05:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.159551.log.gz 2023-11-25T05:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89015.log 2023-11-25T05:36:09.298 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91509.log: /var/log/ceph/ceph-client.admin.92758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93106.log 2023-11-25T05:36:09.298 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.159256.log.gz 2023-11-25T05:36:09.298 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.89015.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91509.log.gz 2023-11-25T05:36:09.298 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93509.log 2023-11-25T05:36:09.298 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91748.log 2023-11-25T05:36:09.298 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.89015.log.gz 2023-11-25T05:36:09.298 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110487.log 2023-11-25T05:36:09.298 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159298.log 2023-11-25T05:36:09.298 INFO:teuthology.orchestra.run.smithi007.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.129425.log.gz 2023-11-25T05:36:09.298 INFO:teuthology.orchestra.run.smithi007.stderr: 91.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159172.log 2023-11-25T05:36:09.298 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.110487.log: -- replaced with /var/log/ceph/ceph-client.admin.92758.log.gz 2023-11-25T05:36:09.298 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.159298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87298.log 2023-11-25T05:36:09.298 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91748.log.gz 2023-11-25T05:36:09.298 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110487.log.gz 2023-11-25T05:36:09.299 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158962.log 2023-11-25T05:36:09.299 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.159172.log: 87.2%gzip -5/var/log/ceph/ceph-client.admin.87298.log: --verbose -- /var/log/ceph/ceph-client.admin.91376.log 2023-11-25T05:36:09.299 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.159298.log.gz 2023-11-25T05:36:09.299 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.158962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111300.log 2023-11-25T05:36:09.299 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.159172.log.gz 2023-11-25T05:36:09.299 INFO:teuthology.orchestra.run.smithi007.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87298.log.gz 2023-11-25T05:36:09.299 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92361.log 2023-11-25T05:36:09.299 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91376.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.158962.log.gz 2023-11-25T05:36:09.299 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.111300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159466.log 2023-11-25T05:36:09.299 INFO:teuthology.orchestra.run.smithi007.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91376.log.gz 2023-11-25T05:36:09.299 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93494.log 2023-11-25T05:36:09.299 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.159466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110497.log 2023-11-25T05:36:09.299 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.92361.log.gz 2023-11-25T05:36:09.299 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose 88.9% -- /var/log/ceph/ceph-client.admin.93510.log -- replaced with /var/log/ceph/ceph-client.admin.159466.log.gz 2023-11-25T05:36:09.300 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:36:09.300 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93494.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.93509.log.gz 2023-11-25T05:36:09.300 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.110497.log: gzip 90.5% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.93106.log.gz -- 2023-11-25T05:36:09.300 INFO:teuthology.orchestra.run.smithi007.stderr: /var/log/ceph/ceph-client.admin.90727.log 2023-11-25T05:36:09.300 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93510.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-25T05:36:09.300 INFO:teuthology.orchestra.run.smithi007.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.111300.log.gz/var/log/ceph/ceph-client.admin.90727.log: 2023-11-25T05:36:09.300 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93098.log 2023-11-25T05:36:09.300 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90688.log 2023-11-25T05:36:09.300 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90727.log.gz 2023-11-25T05:36:09.300 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159046.log 2023-11-25T05:36:09.300 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.90688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91140.log 2023-11-25T05:36:09.300 INFO:teuthology.orchestra.run.smithi007.stderr: 91.2%/var/log/ceph/ceph-client.admin.159046.log: -- replaced with /var/log/ceph/ceph-client.admin.110497.log.gz 2023-11-25T05:36:09.300 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90688.log.gz 2023-11-25T05:36:09.301 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158878.log 2023-11-25T05:36:09.301 INFO:teuthology.orchestra.run.smithi007.stderr: 87.1%/var/log/ceph/ceph-client.admin.91140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92041.log 2023-11-25T05:36:09.301 INFO:teuthology.orchestra.run.smithi007.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.93510.log.gz 2023-11-25T05:36:09.301 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.159046.log.gz 2023-11-25T05:36:09.301 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.158878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110486.log 2023-11-25T05:36:09.301 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91147.log 2023-11-25T05:36:09.301 INFO:teuthology.orchestra.run.smithi007.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91140.log.gz 2023-11-25T05:36:09.301 INFO:teuthology.orchestra.run.smithi007.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.93494.log.gz 2023-11-25T05:36:09.301 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158878.log.gz 2023-11-25T05:36:09.301 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.110486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92893.log 2023-11-25T05:36:09.301 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.92041.log.gz 2023-11-25T05:36:09.301 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90881.log 2023-11-25T05:36:09.301 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91147.log: /var/log/ceph/ceph-client.admin.92893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142327.log 2023-11-25T05:36:09.301 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.90881.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91147.log.gz 2023-11-25T05:36:09.302 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92746.log 2023-11-25T05:36:09.302 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90881.log.gz 2023-11-25T05:36:09.302 INFO:teuthology.orchestra.run.smithi007.stderr: --verbose -- /var/log/ceph/ceph-client.admin.90787.log 2023-11-25T05:36:09.302 INFO:teuthology.orchestra.run.smithi007.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.93098.log.gz 2023-11-25T05:36:09.302 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92746.log: /var/log/ceph/ceph-client.admin.142327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93366.log 2023-11-25T05:36:09.302 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.90787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92101.log 2023-11-25T05:36:09.302 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93366.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90787.log.gz 2023-11-25T05:36:09.302 INFO:teuthology.orchestra.run.smithi007.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.110486.log.gz 2023-11-25T05:36:09.302 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92189.log 2023-11-25T05:36:09.302 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93469.log 2023-11-25T05:36:09.302 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92189.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92101.log.gz 2023-11-25T05:36:09.302 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129427.log 2023-11-25T05:36:09.302 INFO:teuthology.orchestra.run.smithi007.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.142327.log.gz 2023-11-25T05:36:09.302 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92189.log.gz 2023-11-25T05:36:09.303 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93058.log 2023-11-25T05:36:09.303 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92134.log 2023-11-25T05:36:09.303 INFO:teuthology.orchestra.run.smithi007.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.93469.log.gz 2023-11-25T05:36:09.303 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.129427.log: /var/log/ceph/ceph-client.admin.93058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91384.log 2023-11-25T05:36:09.303 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92134.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92134.log.gz 2023-11-25T05:36:09.303 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-25T05:36:09.318 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-11-25T05:36:09.318 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2023-11-25T05:36:09.318 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2023-11-25T05:36:09.318 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2023-11-25T05:36:09.318 INFO:teuthology.orchestra.run.smithi090.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2023-11-25T05:36:09.318 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2023-11-25T05:36:09.319 INFO:teuthology.orchestra.run.smithi090.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2023-11-25T05:36:09.319 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2023-11-25T05:36:09.319 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/valgrind/osd.6.log: /var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86492.log 2023-11-25T05:36:09.319 INFO:teuthology.orchestra.run.smithi090.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2023-11-25T05:36:09.319 INFO:teuthology.orchestra.run.smithi090.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2023-11-25T05:36:09.319 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86412.log 2023-11-25T05:36:09.319 INFO:teuthology.orchestra.run.smithi090.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2023-11-25T05:36:09.319 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.86492.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-25T05:36:09.319 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.86412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86663.log 2023-11-25T05:36:09.319 INFO:teuthology.orchestra.run.smithi090.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86492.log.gz 2023-11-25T05:36:09.319 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86372.log 2023-11-25T05:36:09.319 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86412.log.gz 2023-11-25T05:36:09.319 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-11-25T05:36:09.319 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.86663.log: /var/log/ceph/ceph-client.admin.86372.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-11-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86663.log.gz 2023-11-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 87.9% 2023-11-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86372.log.gz 2023-11-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86452.log 2023-11-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86623.log 2023-11-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.86452.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.86623.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86452.log.gz 2023-11-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86576.log 2023-11-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi090.stderr: 87.2%/var/log/ceph/ceph-mgr.x.log: -- replaced with /var/log/ceph/ceph-client.admin.86623.log.gz 2023-11-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-11-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.86576.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86576.log.gz 2023-11-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86532.log 2023-11-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mon.b.log: 92.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.86532.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.86532.log.gz 2023-11-25T05:36:09.321 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-25T05:36:09.333 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91384.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.91384.log.gz 2023-11-25T05:36:09.333 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92898.log 2023-11-25T05:36:09.333 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.1.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.129427.log.gz 2023-11-25T05:36:09.333 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92842.log 2023-11-25T05:36:09.333 INFO:teuthology.orchestra.run.smithi007.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.93366.log.gz 2023-11-25T05:36:09.333 INFO:teuthology.orchestra.run.smithi007.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.92746.log.gz 2023-11-25T05:36:09.333 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119665.log 2023-11-25T05:36:09.333 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92726.log 2023-11-25T05:36:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.119665.log: 95.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-25T05:36:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91506.log 2023-11-25T05:36:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92726.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.119665.log.gz 2023-11-25T05:36:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129423.log 2023-11-25T05:36:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91506.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91506.log.gz 2023-11-25T05:36:09.334 INFO:teuthology.orchestra.run.smithi007.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.92842.log.gz 2023-11-25T05:36:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91724.log 2023-11-25T05:36:09.334 INFO:teuthology.orchestra.run.smithi007.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.93058.log.gz 2023-11-25T05:36:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.129423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158920.log 2023-11-25T05:36:09.334 INFO:teuthology.orchestra.run.smithi007.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.92898.log.gz 2023-11-25T05:36:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91149.log 2023-11-25T05:36:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.158920.log: /var/log/ceph/ceph-client.admin.87164.log 2023-11-25T05:36:09.334 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91724.log.gz 2023-11-25T05:36:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91149.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158920.log.gz 2023-11-25T05:36:09.335 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.90835.log 2023-11-25T05:36:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.87164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93301.log 2023-11-25T05:36:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.90835.log: 87.8% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.87164.log.gz 2023-11-25T05:36:09.335 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91149.log.gz 2023-11-25T05:36:09.335 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158752.log 2023-11-25T05:36:09.335 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90835.log.gz 2023-11-25T05:36:09.335 INFO:teuthology.orchestra.run.smithi007.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.129423.log.gz 2023-11-25T05:36:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159088.log 2023-11-25T05:36:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93409.log 2023-11-25T05:36:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.158752.log: /var/log/ceph/ceph-client.admin.159088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159340.log 2023-11-25T05:36:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93409.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158752.log.gz 2023-11-25T05:36:09.335 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.159088.log.gz 2023-11-25T05:36:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159004.log 91.2% -- replaced with /var/log/ceph/ceph-client.admin.92726.log.gz 2023-11-25T05:36:09.336 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:36:09.336 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.159340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91799.log 2023-11-25T05:36:09.336 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.159004.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142326.log 2023-11-25T05:36:09.336 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.159340.log.gz 2023-11-25T05:36:09.336 INFO:teuthology.orchestra.run.smithi007.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.93409.log.gz 2023-11-25T05:36:09.336 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92319.log 2023-11-25T05:36:09.336 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91799.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.159004.log.gz 2023-11-25T05:36:09.336 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.142326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92709.log 2023-11-25T05:36:09.336 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92319.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91799.log.gz 2023-11-25T05:36:09.336 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-11-25T05:36:09.336 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92709.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.92319.log.gz 2023-11-25T05:36:09.336 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.91148.log 2023-11-25T05:36:09.336 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88031.log 2023-11-25T05:36:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91148.log: /var/log/ceph/ceph-osd.3.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.142326.log.gz 2023-11-25T05:36:09.337 INFO:teuthology.orchestra.run.smithi007.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91148.log.gz 2023-11-25T05:36:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119662.log 2023-11-25T05:36:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.88031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93495.log 2023-11-25T05:36:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.119662.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88031.log.gz 2023-11-25T05:36:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93181.log 2023-11-25T05:36:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93495.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.93495.log.gz 2023-11-25T05:36:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159382.log 2023-11-25T05:36:09.337 INFO:teuthology.orchestra.run.smithi007.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.93301.log.gz 2023-11-25T05:36:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93181.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.119662.log.gz 2023-11-25T05:36:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91378.log 2023-11-25T05:36:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.159382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119664.log 2023-11-25T05:36:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91378.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.159382.log.gz 2023-11-25T05:36:09.337 INFO:teuthology.orchestra.run.smithi007.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91378.log.gz 2023-11-25T05:36:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92773.log 2023-11-25T05:36:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.119664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129426.log 2023-11-25T05:36:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92773.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.119664.log.gz 2023-11-25T05:36:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159508.log 2023-11-25T05:36:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.129426.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.93181.log.gz 2023-11-25T05:36:09.338 INFO:teuthology.orchestra.run.smithi007.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.129426.log.gz 2023-11-25T05:36:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110495.log 2023-11-25T05:36:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.159508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92688.log 2023-11-25T05:36:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.110495.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.159508.log.gz 2023-11-25T05:36:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158668.log 2023-11-25T05:36:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92688.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.110495.log.gz 2023-11-25T05:36:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93164.log 2023-11-25T05:36:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.158668.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.158668.log.gz 2023-11-25T05:36:09.339 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-11-25T05:36:09.339 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93164.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.93164.log.gz 2023-11-25T05:36:09.339 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91433.log 2023-11-25T05:36:09.346 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159593.log 2023-11-25T05:36:09.347 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91433.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.91433.log.gz 2023-11-25T05:36:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91413.log 2023-11-25T05:36:09.359 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.159593.log: 87.9% 89.9% -- replaced with /var/log/ceph/ceph-client.admin.92688.log.gz 2023-11-25T05:36:09.359 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.159593.log.gz 2023-11-25T05:36:09.359 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159130.log 2023-11-25T05:36:09.362 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91413.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91413.log.gz 2023-11-25T05:36:09.362 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91139.log 2023-11-25T05:36:09.362 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.159130.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.159130.log.gz 2023-11-25T05:36:09.363 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-25T05:36:09.363 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91139.log: 94.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-11-25T05:36:09.363 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91139.log.gz 2023-11-25T05:36:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111299.log 2023-11-25T05:36:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93177.log 2023-11-25T05:36:09.366 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.111299.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.111299.log.gz 2023-11-25T05:36:09.367 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92835.log 2023-11-25T05:36:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93177.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.93177.log.gz 2023-11-25T05:36:09.376 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93467.log 2023-11-25T05:36:09.376 INFO:teuthology.orchestra.run.smithi007.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.92709.log.gz 2023-11-25T05:36:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92835.log: 94.3% -- replaced with /var/log/ceph/ceph-client.admin.92893.log.gz 2023-11-25T05:36:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92934.log 2023-11-25T05:36:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93021.log 2023-11-25T05:36:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92934.log: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.92773.log.gz 2023-11-25T05:36:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-25T05:36:09.383 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93021.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.93467.log.gz 2023-11-25T05:36:09.383 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91619.log 2023-11-25T05:36:09.385 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.92934.log.gz 2023-11-25T05:36:09.386 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-25T05:36:09.387 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91619.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91619.log.gz 2023-11-25T05:36:09.387 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91150.log 2023-11-25T05:36:09.389 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.2.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.93021.log.gz 2023-11-25T05:36:09.389 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90305.log 2023-11-25T05:36:09.390 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91150.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91150.log.gz 2023-11-25T05:36:09.390 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158710.log 2023-11-25T05:36:09.391 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.90305.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90305.log.gz 2023-11-25T05:36:09.392 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91144.log 2023-11-25T05:36:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.158710.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.158710.log.gz 2023-11-25T05:36:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92908.log 2023-11-25T05:36:09.394 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91144.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91144.log.gz 2023-11-25T05:36:09.394 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93350.log 2023-11-25T05:36:09.394 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92908.log: 93.2% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-25T05:36:09.395 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129424.log 2023-11-25T05:36:09.408 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91957.log 2023-11-25T05:36:09.412 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.129424.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.129424.log.gz 2023-11-25T05:36:09.421 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159633.log 2023-11-25T05:36:09.422 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91957.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91957.log.gz 2023-11-25T05:36:09.440 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91685.log 2023-11-25T05:36:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.159633.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.159633.log.gz 2023-11-25T05:36:09.449 INFO:teuthology.orchestra.run.smithi007.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.93350.log.gz 2023-11-25T05:36:09.450 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92273.log 2023-11-25T05:36:09.451 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91685.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91685.log.gz 2023-11-25T05:36:09.452 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111301.log 2023-11-25T05:36:09.453 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92273.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92273.log.gz 2023-11-25T05:36:09.453 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91153.log 2023-11-25T05:36:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.111301.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.111301.log.gz 2023-11-25T05:36:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90925.log 2023-11-25T05:36:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.91153.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91153.log.gz 2023-11-25T05:36:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158836.log 2023-11-25T05:36:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.90925.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90925.log.gz 2023-11-25T05:36:09.459 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92403.log 2023-11-25T05:36:09.461 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.158836.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.158836.log.gz 2023-11-25T05:36:09.461 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158794.log 2023-11-25T05:36:09.461 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.92403.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.92403.log.gz 2023-11-25T05:36:09.463 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93250.log 2023-11-25T05:36:09.463 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.158794.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158794.log.gz 2023-11-25T05:36:09.463 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-11-25T05:36:09.470 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.93250.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.93250.log.gz 2023-11-25T05:36:09.542 INFO:teuthology.orchestra.run.smithi090.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-25T05:36:09.937 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mon.c.log: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.92908.log.gz 2023-11-25T05:36:10.716 INFO:teuthology.orchestra.run.smithi007.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-11-25T05:36:10.764 INFO:teuthology.orchestra.run.smithi007.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92835.log.gz 2023-11-25T05:36:11.177 INFO:teuthology.orchestra.run.smithi090.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-11-25T05:36:12.360 INFO:teuthology.orchestra.run.smithi007.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-25T05:36:12.858 INFO:teuthology.orchestra.run.smithi090.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-11-25T05:36:13.118 INFO:teuthology.orchestra.run.smithi090.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-11-25T05:36:13.327 INFO:teuthology.orchestra.run.smithi090.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-11-25T05:36:13.497 INFO:teuthology.orchestra.run.smithi007.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-25T05:36:13.531 INFO:teuthology.orchestra.run.smithi007.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-25T05:36:13.780 INFO:teuthology.orchestra.run.smithi007.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-11-25T05:36:13.986 INFO:teuthology.orchestra.run.smithi007.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-25T05:36:13.989 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-25T05:36:13.989 INFO:teuthology.orchestra.run.smithi007.stderr:real 0m4.735s 2023-11-25T05:36:13.989 INFO:teuthology.orchestra.run.smithi007.stderr:user 0m23.534s 2023-11-25T05:36:13.989 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m0.810s 2023-11-25T05:36:14.018 INFO:teuthology.orchestra.run.smithi090.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-11-25T05:36:14.019 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-25T05:36:14.019 INFO:teuthology.orchestra.run.smithi090.stderr:real 0m4.730s 2023-11-25T05:36:14.020 INFO:teuthology.orchestra.run.smithi090.stderr:user 0m17.624s 2023-11-25T05:36:14.020 INFO:teuthology.orchestra.run.smithi090.stderr:sys 0m0.608s 2023-11-25T05:36:14.020 INFO:tasks.ceph:Archiving logs... 2023-11-25T05:36:14.020 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196/remote/smithi007/log 2023-11-25T05:36:14.021 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-25T05:36:17.065 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196/remote/smithi090/log 2023-11-25T05:36:17.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-25T05:36:19.577 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ec928368edcc5fc71884a56fd9fd821b148f4e30/qa/tasks/ceph.py", line 1957, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ec928368edcc5fc71884a56fd9fd821b148f4e30/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ec928368edcc5fc71884a56fd9fd821b148f4e30/qa/tasks/ceph.py", line 368, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-11-25T05:36:19.578 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-25T05:36:19.587 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ec928368edcc5fc71884a56fd9fd821b148f4e30/qa/tasks/ceph.py", line 1957, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ec928368edcc5fc71884a56fd9fd821b148f4e30/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ec928368edcc5fc71884a56fd9fd821b148f4e30/qa/tasks/ceph.py", line 368, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-11-25T05:36:19.587 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-25T05:36:19.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-25T05:36:19.590 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-25T05:36:19.679 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-25T05:36:19.687 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-25T05:36:19.687 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-25T05:36:19.690 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-25T05:36:19.704 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2023-11-25T05:36:19.706 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2023-11-25T05:36:19.709 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-25T05:36:19.709 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2023-11-25T05:36:19.709 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 +785us[ +785us] +/- 94ms 2023-11-25T05:36:19.709 INFO:teuthology.orchestra.run.smithi007.stdout:^* hv02.front.sepia.ceph.com 3 6 377 63 -331us[ -346us] +/- 66ms 2023-11-25T05:36:19.710 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 64 -195us[ -210us] +/- 79ms 2023-11-25T05:36:19.710 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-25T05:36:19.711 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-25T05:36:19.711 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2023-11-25T05:36:19.711 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 +761us[ +770us] +/- 94ms 2023-11-25T05:36:19.711 INFO:teuthology.orchestra.run.smithi090.stdout:^* hv02.front.sepia.ceph.com 3 6 377 0 -349us[ -337us] +/- 67ms 2023-11-25T05:36:19.711 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -212us[ -203us] +/- 79ms 2023-11-25T05:36:19.711 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-25T05:36:19.712 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-25T05:36:19.720 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-25T05:36:19.721 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-25T05:36:19.743 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-25T05:36:19.861 DEBUG:teuthology.orchestra.run.smithi090:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-25T05:36:19.960 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-25T05:36:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-25T05:36:19.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-25T05:36:20.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-25T05:36:20.033 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-25T05:36:20.042 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-25T05:36:20.061 INFO:teuthology.task.internal:Duration was 2948.148344 seconds 2023-11-25T05:36:20.061 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-25T05:36:20.094 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-25T05:36:20.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-25T05:36:20.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-25T05:36:20.141 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-25T05:36:20.147 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-25T05:36:20.300 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-25T05:36:20.300 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2023-11-25T05:36:20.300 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-25T05:36:20.320 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2023-11-25T05:36:20.321 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-25T05:36:20.345 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-25T05:36:20.346 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-25T05:36:20.364 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-25T05:36:20.415 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-25T05:36:20.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-25T05:36:20.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-25T05:36:20.551 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-25T05:36:20.561 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-25T05:36:20.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-25T05:36:20.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-25T05:36:20.620 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-25T05:36:20.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-25T05:36:20.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-25T05:36:20.662 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2023-11-25T05:36:20.685 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2023-11-25T05:36:20.705 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-25T05:36:20.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-25T05:36:20.737 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-25T05:36:20.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-25T05:36:20.759 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-25T05:36:20.769 INFO:teuthology.task.internal:Transferring archived files... 2023-11-25T05:36:20.770 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196/remote/smithi007 2023-11-25T05:36:20.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-25T05:36:20.818 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196/remote/smithi090 2023-11-25T05:36:20.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-25T05:36:20.859 INFO:teuthology.task.internal:Removing archive directory... 2023-11-25T05:36:20.878 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-25T05:36:20.881 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-25T05:36:20.909 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-25T05:36:20.972 INFO:teuthology.task.internal:Not uploading archives. 2023-11-25T05:36:20.972 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-25T05:36:20.984 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-25T05:36:20.984 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-25T05:36:20.987 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-25T05:36:20.999 INFO:teuthology.orchestra.run.smithi007.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 25 05:36 /home/ubuntu/cephtest 2023-11-25T05:36:21.002 INFO:teuthology.orchestra.run.smithi090.stdout: 265884 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 25 05:36 /home/ubuntu/cephtest 2023-11-25T05:36:21.003 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-25T05:36:21.015 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-25T05:36:21.124 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: saw valgrind issues 2023-11-25T05:36:21.124 INFO:teuthology.nuke:Checking targets against current locks 2023-11-25T05:36:21.150 DEBUG:teuthology.nuke:shortname: smithi007 2023-11-25T05:36:21.150 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-25T05:36:21.173 DEBUG:teuthology.nuke:shortname: smithi090 2023-11-25T05:36:21.173 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-25T05:36:21.176 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-25 04:37:47.713884', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIQLzUj5DoDXok6ckR54zrIrE0YNr5/E8OHdWoHANIrAr9UBNH2fLFLMlj9n4BT4u5otCq4Gq1yMp/4oCqYCggY='} 2023-11-25T05:36:21.195 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-24_21:18:30-rados-main-distro-default-smithi/7466196', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-25 04:37:47.715842', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFtIGrbVuXwbnaMq4SVU04AhZvexOMXUoe+9Sjmpom43axHDTqNK2H6t0ELoI7LvcVNKRAUXDUS45FAfH/4JmEE='} 2023-11-25T05:36:21.223 INFO:teuthology.orchestra.console.smithi007:Power off 2023-11-25T05:36:21.224 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-25T05:36:21.248 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2023-11-25T05:36:21.249 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-25T05:36:21.264 INFO:teuthology.orchestra.console.smithi090:Power off 2023-11-25T05:36:21.265 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-25T05:36:21.274 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2023-11-25T05:36:21.287 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off 2023-11-25T05:36:21.288 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-25T05:36:21.312 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2023-11-25T05:36:25.275 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-25T05:36:25.313 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-25T05:36:25.385 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2023-11-25T05:36:25.423 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2023-11-25T05:36:29.386 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-25T05:36:29.424 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-25T05:36:29.497 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2023-11-25T05:36:29.538 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2023-11-25T05:36:29.599 INFO:teuthology.orchestra.console.smithi007:Power off completed 2023-11-25T05:36:33.542 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-25T05:36:33.651 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off 2023-11-25T05:36:33.752 INFO:teuthology.orchestra.console.smithi090:Power off completed 2023-11-25T05:36:33.856 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 2948.1483442783356 failure_reason: saw valgrind issues flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2023-11-25T05:36:33.856 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-25T05:36:33.962 INFO:teuthology.run:FAIL